{"id":2529,"date":"2015-08-05T02:29:35","date_gmt":"2015-08-05T02:29:35","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=2529"},"modified":"2020-03-30T02:43:21","modified_gmt":"2020-03-30T02:43:21","slug":"migrating-a-sql-server-database-from-appharbor-to-amazon-rds","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/migrating-a-sql-server-database-from-appharbor-to-amazon-rds\/","title":{"rendered":"Migrating a SQL Server Database from AppHarbor to Amazon RDS"},"content":{"rendered":"<p>SQL Server Management studio provides a tool to migrate data between two different databases. In my testing I found that it  does not copy constraints, but it is fairly fast to transfer data, as it copies tables in parallel. To resolve that, I&#8217;ve found that using Liquibase to build the new database first, and SQL Server Management Studio to transfer data works fairly well.<\/p>\n<p>Preparation steps:<br \/>\n1. If using an AWS reseller, determine what datacenter they put the database in<br \/>\n2. Determine which version of SQL Server the original database is<\/p>\n<pre lang=\"sql\">select @@VERSION\n<\/pre>\n<p>Microsoft SQL Server 2008 R2 (RTM) &#8211; 10.50.1600.1 (Intel X86)   Apr  2 2010 15:53:02   Copyright (c) Microsoft Corporation  Express Edition with Advanced Services on Windows NT 6.0 <x86> (Build 6002: Service Pack 2) (Hypervisor) <\/x86><\/p>\n<p>3. Determine how much disk space your database uses. For this you can attempt a <a href=\"http:\/\/blog.sqlauthority.com\/2013\/08\/02\/sql-server-disk-space-monitoring-detecting-low-disk-space-on-server\/\">query<\/a>, but you may have an easier time consulting the administration website for your existing database.<\/p>\n<p>4. Create a new database in RDS within the same datacenter, using the version number found in step 2.<\/p>\n<p>5. Take a database backup of the original. Unfortunately SQL server is terrible in shared environments, so your best bet may be <a href=\"https:\/\/sqlbackupandftp.com\/\">SQL Backup and FTP<\/a>, which is difficult to restore from.<\/p>\n<p>6. <a href=\"https:\/\/www.itsupportguides.com\/server-side-tips\/sql-management-studio-how-to-create-read-only-users\/\">Create a readonly account<\/a> in the source database to work with (this prevents accidental modifications)<\/p>\n<p>7. Check which tables are the largest &#8211; if you have tables containing audit history or logs, you may be surprised at their size. If you want to make the migration faster, you&#8217;ll need to delete old entries.<\/p>\n<p>If you use autoincrement fields, you will also need this information, because you will need to set the starting values in the new database on creation.<\/p>\n<pre lang=\"sql\">SELECT\n    sysobjects.*\n    , sysindexes.ROWS\nFROM\n    sysobjects\n    INNER JOIN sysindexes\n    ON sysobjects.id = sysindexes.id\nWHERE\n    TYPE = 'U'\n    AND sysindexes.IndId &lt; 2\nORDER BY sysindexes.ROWS DESC\n<\/pre>\n<p>8. Download <a href=\"http:\/\/www.liquibase.org\/\">Liquibase<\/a> and the SQL Server <a href=\"https:\/\/msdn.microsoft.com\/en-us\/sqlserver\/aa937724.aspx\">jdbc driver<\/a>.<\/p>\n<p>9. Create a configuration fil to point to both databases:<\/p>\n<pre>classpath=D:\\\\tools\\\\sqljdbc4.jar\ndriver=com.microsoft.sqlserver.jdbc.SQLServerDriver\nurl=jdbc:sqlserver:\/\/x.us-east-1.rds.amazonaws.com:1433;databaseName=db\nusername=\npassword=\nchangeLogFile=D:\\\\tools\\\\liquibase\\\\ChangeLog.xml\n\n#referenceUrl=jdbc:sqlserver:\/\/x.sqlserver.sequelizer.com\nreferenceUsername=\nreferencePassword=\n<\/pre>\n<p>10. Have liquibase diff the &#8220;old&#8221; and &#8220;new&#8221; databases. This compares the two schemas (one being empty) and tells you how much work it&#8217;s going to do when you actually run it.<\/p>\n<pre lang=\"bash\">liquibase diff\n<\/pre>\n<pre>Unexpected Index(s): NONE\nChanged Index(s): NONE\nMissing Primary Key(s):\n     PK_ABExperiments_1 on ABExperiments(UniqueId, Experiment)\n     PK_AccountCredits on AccountCredits(AccountId)\nUnexpected Primary Key(s): NONE\nChanged Primary Key(s): NONE\nMissing Schema(s): NONE\nUnexpected Schema(s): NONE\nChanged Schema(s): NONE\nMissing Sequence(s): NONE\nUnexpected Sequence(s): NONE\nChanged Sequence(s): NONE\nMissing Stored Procedure(s): NONE\nUnexpected Stored Procedure(s): NONE\nChanged Stored Procedure(s): NONE\nMissing Table(s): NONE\nUnexpected Table(s): NONE\nChanged Table(s): NONE\nMissing Unique Constraint(s): NONE\nUnexpected Unique Constraint(s): NONE\nChanged Unique Constraint(s): NONE\n<\/pre>\n<p>11. Have Liquibase generate a change log:<\/p>\n<pre lang=\"bash\">liquibase diffChangeLog\n<\/pre>\n<p>This produces an XML file with all the changes required to produce the new database. You will likely need to edit this file.<\/p>\n<p>12. You can attempt to run the change log, to determine what problems you&#8217;ll hit, but running:<\/p>\n<pre lang=\"bash\">liquibase update\n<\/pre>\n<p>13. If you use autoincrement fields, you will need to set the starting values. I picked a value that was larger than any of mine, because it&#8217;s easier to find-replace in this file:<\/p>\n<pre lang=\"xml\"><column autoincrement=\"true\" startwith=\"150000\" name=\"Id\" type=\"int\">\n<\/column><\/pre>\n<p>14. You will likely need to re-create the database several times to get this right. There are a few types it struggles with (e.g. blob types) so you may want to watch for these and change them in the source system.<\/p>\n<p>15. Before running a data migration, you need to disable foreign keys.<\/p>\n<pre lang=\"sql\">EXEC sp_msforeachtable \"ALTER TABLE ? NOCHECK CONSTRAINT all\"\n<\/pre>\n<p>16. You can then suffer through the SQL Server UI.<\/p>\n<p>Select &#8220;SQL Server Authentication&#8221;:<\/p>\n<p>17. For each table you have to edit the migration settings and check &#8220;Set insert on&#8221; for each table. This setting is designed for migrations, because you can only set it on one table at a time &#8211; it allows you to update the values of &#8220;identity&#8221; columns.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2015\/08\/1-578x480.png\" alt=\"1\" width=\"578\" height=\"480\" class=\"aligncenter size-large wp-image-2534\"><\/p>\n<p>18. Run the data migration and wait.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/172.104.26.128\/wp-content\/uploads\/2015\/08\/3.png\" alt=\"3\" width=\"566\" height=\"579\" class=\"aligncenter size-full wp-image-2535\" srcset=\"https:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2015\/08\/3.png 566w, https:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2015\/08\/3-293x300.png 293w\" sizes=\"(max-width: 566px) 100vw, 566px\" \/><\/p>\n<p>19. Remove the tables that Liquibase creates (unless you need these)<\/p>\n<pre lang=\"sql\">drop table dbo.DATABASAECHANGELOG;\ndrop table dbo.DATABASECHANGELOGLOCK;\n<\/pre>\n<p>20. Re-enable all the constraints:<\/p>\n<pre lang=\"sql\">EXEC sp_msforeachtable \"ALTER TABLE ? CHECK CONSTRAINT all\"\n<\/pre>\n<p>21. Verify that the application works.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>SQL Server Management studio provides a tool to migrate data between two different databases. In my testing I found that it does not copy constraints, but it is fairly fast to transfer data, as it copies tables in parallel. To resolve that, I&#8217;ve found that using Liquibase to build the new database first, and SQL &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.garysieling.com\/blog\/migrating-a-sql-server-database-from-appharbor-to-amazon-rds\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Migrating a SQL Server Database from AppHarbor to Amazon RDS&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[4,8],"tags":[160,204,367,524],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/2529"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=2529"}],"version-history":[{"count":1,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/2529\/revisions"}],"predecessor-version":[{"id":6471,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/2529\/revisions\/6471"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=2529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=2529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=2529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}