Categories
Design

How to sync your kodi library across multiple devices with mysql

I run Kodi on a Windows 10 machine, ATV4, Fire TV and an Android box. Is there anyway to easily sync the setups from my main Kodi (Win10 machine) to all the other Kodis? I'm getting tired of configuring and making changes to all boxes and want them all to mirror each other.

You'll have to install MySQL on a computer, then put in the settings in advancedsettings.xml for database sync.

It really only covers watched status, but I've recently become a pretty big fan of the program addon called trakt, which keeps track of watch status even if you lose all your settings. Even if you use a central server like emby or plex, trakt might be nice to have installed, as it works independent of local servers.

If you do install it, I'd suggest turning off auto ratings though. Lot of annoying hassle to leave ratings going.

I do use Trakt. I'm a huge fan!

Check the wiki. It's all laid out there. You want a mysql database.

I've got mine setup with a shared mysql library on a Synology NAS (videos on NAS as well). 2 Kodi installs read from it (both are nVididia Shields). It works great, add/scan a video on one install and the other sees it automatically. Likewise, start watching a video on one and resume on the other, etc. I'm planning on adding another Shield in the future, all I'll have to do is drop a couple of config files onto the device and I'm good to go. I only have one very small complaint – there is just a bit of lag in terms of cover art appearing and such. I wouldn't even have noticed it if I hadn't been running both devices with local installs to begin with (db has

5K videos). Mind you, I'm being really nit-picky here, everything else is fabulous.

When using a Kodi MySQL database shared library, sync Genesis libraries across HTPCs by adding the video sources using Windows network (SMB) network paths .

How to sync your kodi library across multiple devices with mysql

This is a follow-up to my previous post, How to Merge Kodi Streaming Add-ons into the Video Library. If you haven’t read that post, please check that out first. After many years of being an XBMC / Kodi user, it was a complete game changer.

Now, after I set up my main home theater PC (HTPC) with TV shows and movies merged into the library, I noticed an issue where other HTPCs in my home couldn’t access that media. Episodes would appear, but the links were broken.

Kodi MySQL Database Library Sharing

Before I go any further, I should probably explain why and how my movies and TV shows appear on other computers. The reason is because I setup a MySQL shared library.

You see, on my “main” HTPC, I installed MySQL server, and configured every instance of Kodi in my house to look to that MySQL database for the library. You do that by adding an “advancedsettings.xml” file to your userdata folder (For Windows, located in C:\Users\YourUsername\AppData\Roaming\Kodi\userdata\). The contents of that xml files should look something like this (swapping the highlighted parts with your own settings).

With this setup, every Kodi HTPC in your house shares the exact same library. You can watch half of a movie in one room, and pick up from that same spot in another room. Watched statuses get synced across all computers.

Don’t Add a Source with a Local Path When Using a MySQL Database Shared Library

Getting back to the issue I was talking about earlier, based on my previous post, the TV shows and movies in my library that were from Genesis would only work on the main computer because I had added the sources using a local path like…

Use Windows network (SMB) Paths for Shared Sources

The solution to sync Genesis libraries across all of my HTPCs was to add a new source that basically pointed to the same folder but using the network path using Windows network (SMB).

How to sync your kodi library across multiple devices with mysql

Of course, you will need to make sure you share that folder to make it available over the network. In my case, I just shared my entire Users folder, so you can see in the below screenshot, I had to drill all the way down to that same folder.

How to sync your kodi library across multiple devices with mysql

After adding the new source using the network path, I then attempted to mark shows, seasons, and episodes as watched so the network source roughly matched the local source in terms of watched status. During this state, all episodes show twice. Once from the local source and another from the network source.

When that was done, I removed the local source leaving only the network source. I then tested on one of my other HTPCs and the media loaded up exactly the same as the main HTPC.

Enjoy! If you have any questions or concerns, be sure to Ask a Question and I would love to help you out. Thanks!

I have run Kodi on a Rasperry Pi 1 for the past 8 years and this week have decided to replace it with a new 2GB Pi4.

My media is stored on a Synology NAS (a DS210j originally and then, following a recent upgrade, a DS218+) and is accessed using NFS.

Because the Kodi implementation I have used previously (OSMC) does not yet support Pi4 I have decided to move to LibreELEC, downloading the Pi4 image of 9.2.0. At the same time I thought I would also consider creating a central library using MySQL so that I can add more LibreELEC installations in future and run all from the same library.

I do not consider myself to be an expert, but I am able to follow instructions without too much difficulty. I installed MySQL server onto another old Pi1 (also used for couple of other basic jobs) and have been able to test a remote connection from a Windows machine (the same machine I am using to type this) using an application called HeidiSQL to confirm that I could connect to the MySQL installation and that the username/password I had created for Kodi to use also worked.

I then found on the Kodi Wiki how to create the file advancedsettings.xml and that this file should be placed in the .kodi/userdata folder. I also copied the sources.xml file from my old Kodi installation so that I did not need to create the media sources from scratch.

In the LibreELEC settings (Network –> Advanced Network Settings) I have enabled “Wait for network before starting Kodi” and set the Maximum Wait Time value to 30 seconds (probably longer than needed even though I am using WiFi).

My advancedsettings.xml file looks like this:-

I restarted so that it could re-read the copied sources.xml file. With this all in place, I then went to Settings –> Media –> Videos. right-clicked on Movies source to configure the scraper settings. Once done, Kodi then prompts me to refresh the information. I select Yes and the system then proceeed to build the library.

I then did the same for my TV media source and left the system to build that library.

From a Kodi perspective the content is all present. I can select any entry from the library and it begins to play (I haven’t tested a full programme/movie as my old Pi is still in place but from a basic playback perspective the new installation seems to be working).

However, when I log into MySQL database and run “show databases;” there is no sign of the new database I would expect to find (just the 3 default databases).

I can only assume therefore that the library has been created locally, but I am unsure why it has not connected to the MySQL server and created the library database there. I have looked at .kodi/temp/kodi.log and can find no references to ‘mysql’ or the IP address of the other Pi running MySQL to indicate an error.

Can anyone advise if any of the steps I have carried our were not correct or let me know if there are any additional logs I can check (and where to find them) to see if I can find out if this LibreELEC/Kodi installation is even trying to connect to MySQL.

The kids have their tablets and the wife an Android TV Box in the bedroom. I’ve tried to centralise everything so now all content is stored on an HTPC which I’ve built over the last few months in the lounge. The catch is I don’t only use it as an HTPC, I play games on the TV as well.

What irritates me is the fact that if Kodi isn’t active/running then none of the other devices can access the media library over uPnP. Perhaps this is an innate limitation of uPnP, I haven’t done any research yet.

Any of you guys have a similar setup? How do you get around this?

Pilgrim

Wugger
  • Aug 26, 2016
  • #2

I have a dedicated fileserver running Kodi with a MySQL database. Most of my other devices run through uPnP but i have played with some other devices sharing the same SQL server. So far uPnP performs best on mobile devices.

Your Kodi instance hosts the uPnP service so closing it does mean no other device can run. You could try starting Kodi under a different user profile so that you don’t see it, but it will use a bit of resources.

bwana

MyBroadband
  • Aug 26, 2016
  • #3

I have an, what I find as annoying, issue.

The kids have their tablets and the wife an Android TV Box in the bedroom. I’ve tried to centralise everything so now all content is stored on an HTPC which I’ve built over the last few months in the lounge. The catch is I don’t only use it as an HTPC, I play games on the TV as well.

What irritates me is the fact that if Kodi isn’t active/running then none of the other devices can access the media library over uPnP. Perhaps this is an innate limitation of uPnP, I haven’t done any research yet.

Any of you guys have a similar setup? How do you get around this?

dualmeister

Honorary Master
  • Aug 26, 2016
  • #4

That was going to be my answer but .

xrapidx

Honorary Master
  • Aug 26, 2016
  • #5

dualmeister

Honorary Master
  • Aug 26, 2016
  • #6

The_Ogre

Honorary Master
  • Aug 26, 2016
  • #7

I’m not against that, I think I’ll install PMS later and then configure the other devices to use it as a source.

I have a dedicated fileserver running Kodi with a MySQL database. Most of my other devices run through uPnP but i have played with some other devices sharing the same SQL server. So far uPnP performs best on mobile devices.

Your Kodi instance hosts the uPnP service so closing it does mean no other device can run. You could try starting Kodi under a different user profile so that you don’t see it, but it will use a bit of resources.

That was going to be my answer but .

Playing games with Plex serving content at the same time . maybe a beefy PC can, but not sure :erm:

The_Ogre

Honorary Master
  • Aug 26, 2016
  • #8

stricken

Expert Member
  • Aug 26, 2016
  • #9

K3NS31

Expert Member
  • Aug 26, 2016
  • #10

The_Ogre

Honorary Master
  • Aug 26, 2016
  • #11

Thanks for the heads up, that will have to wait until later. Right now the machine only uses mechanical drives

dualmeister

Honorary Master
  • Aug 26, 2016
  • #12

I’m not against that, I think I’ll install PMS later and then configure the other devices to use it as a source.

]Its a 4th gen i5 with 8GB of RAM, so not too bad, but I’m not sure how much Kodi running actually chows, I’ll have to check.

dualmeister

Honorary Master
  • Aug 26, 2016
  • #13

stepper

Expert Member
  • Aug 26, 2016
  • #14

The_Ogre

Honorary Master
  • Aug 26, 2016
  • #15

stepper

Expert Member
  • Aug 26, 2016
  • #16

dualmeister

Honorary Master
  • Aug 26, 2016
  • #17

stepper

Expert Member
  • Aug 26, 2016
  • #18

dualmeister

Honorary Master
  • Aug 26, 2016
  • #19

K3NS31

Expert Member
  • Aug 26, 2016
  • #20

Nice comparison – this part is the kicker for me, personally:

"
Well, quite frankly this is where Emby really outshines Plex in the Plex vs Emby comparison. The major advantage of using Emby vs Plex is that Emby, through the Emby for Kodi Add-on, fully integrates itself into the local Kodi media database. This means that Kodi and all of its add-ons acts just as if Kodi had scraped the metadata itself and placed it into the database.

The PleXMBC add-on (Plex for Kodi), on the other hand is a completely independent add-on which doesn’t play well with other add-ons. It connects to and interacts with the Plex server independent of the local Kodi database, and any add-ons not designed for use specifically with Plex will not function correctly as they will be pointing to the local Kodi database rather than the Plex server. There are also only a few skins which are fully Plex integrated.
"

We use Kodi as our main media player, but one thing it’s really lacking is a centralised way of it managing what you’ve watched, paused, etc across multiple devices, and I wanted to try to set it up a bit better so when I watch on one device it’s recorded the ‘watch’ on others, etc.

From what I can see there are three usual methods;
1) syncing to a MySQL database, which I tried a few years ago and found to be a ballache to set up and very sluggish in use
2) using Trakt to log watched shows and movies
3) Using Emby Server and the Emby addons in Kodi and let Emby keep track of whats been watched and paused, etc.

I have been using method 2, which has been fine, but it’s not perfect, as it means every device only sends or picks up the info when it syncs with Trakt, so you can’t really pause on one device and pick up on another.

Is anyone using method 3 successfully? Does it keep sync well? And what about when the Emby server is down? I can see that in Emby Server you can add direct paths for folders as well as the local paths so presumably that info goes over to Kodi so it can play directly? And when it comes to using Trakt (to log watched history) is it best to disable it in Emby and let Kodi do it, or vice versa?

Hope that makes sense.

next010

Distinguished Member
  • Mar 2, 2018
  • #2

In my experience with method 3 using Emby (or Plex) they take care of everything, the Kodi TV/Movie libraries are filled with it’s listings as if it were an Emby client.

Resume function works across clients same as any other client.

Never messed around with direct/local paths for folders I guess these are for offline playback, in normal usage if server goes down then your client is useless.

Stevie G

Active Member
  • Mar 3, 2018
  • #3

Thanks, I think I’ll test it on my laptop to see if it does what I want. I find the plex soon for Kodi pretty useless, it just replicates plex in Kodi and may as well just actually use plex.

The shield can play anything I throw at it without transcoding so I don’t want it reliant on Emby needing to be running as it can occasionally play silly buggers so the direct path option sounds useful

ChrisN

Active Member
  • Mar 3, 2018
  • #4

next010

Distinguished Member
  • Mar 3, 2018
  • #5

MrMC an app store fork of Kodi is also available on the Shield it has it’s own custom UI and Plex integration which integrates the listings directly, there are no addons with MrMC. It also always direct plays content and doesn’t invoke transcoding.

Alternatively the unofficial Plex addon should do the trick assuming it is still maintained for Kodi.

Kodi is unquestionably the most popular media center software available today. The application falls under the open source license which means that it is 100% free and it does not include annoying advertisements.

The true power of Kodi comes from the thousands of various builds & add-ons that can be easily added to the software to provide additional functionality. This Web site includes numerous tutorials that will teach you how to install these builds & addons that will provide free movies, tv shows, sports, live tv, and much more.

The most popular Kodi device is currently the Fire TV and Fire TV Stick due to their low price points and quality internal components. The Fire TV/Stick allows for people to use various streaming applications and IPTV services that provide free movies, tv shows, live channels, etc.

For more information on kodi and what’s the best kodi builds & addons check out our popular list updated Monthly:

Kodi® media center is an award-winning free and open source cross-platform software media player and entertainment hub for digital media for HTPCs (Home theater PCs). It uses a 10-foot user interface designed to be a media player for the living-room, using a remote control as the primary input device. Its graphical user interface (GUI) allows the user to easily browse and view videos, photos, podcasts, and music from a harddrive, optical disc, local network, and the internet using only a few buttons.


The official Kodi version does not contain any content what so ever. This means that you should provide your own content from a local or remote storage location, DVD, Blu-Ray or any other media carrier that you own. Additionally Kodi allows you to install third-party plugins that may provide access to content that is freely available on the official content provider website. Any other means of watching illegal content which would otherwise be paid for is not endorsed or approved by Team Kodi.

Estuary is the new standard skin and is designed to be fast and user friendly.
With the skin Estouchy , Kodi has now been enhanced to better work with larger 5″ or up phones and tablets. Kodi is not designed or recommended for use with smaller phones.

License and development:


Kodi® is a trademark of the XBMC Foundation. For further details you can visit http://kodi.wiki/view/Official:Trademark_Policy

Kodi® is fully Open-Source and released under the GPLv2.0+ license. It includes many third-party libraries which use compatible licenses. Due to the inclusion of some GPLv3.0 libraries the entire application becomes GPLv3.0 as binary.
Should you wish to help on future development you can do so by visiting our forum for further questions.

What Devices Does Kodi Support?

You likely own at least one device that supports Kodi, since this free software is available for Windows, macOS, Linux, Android, iOS, and even Raspberry Pi (RPi) devices. Check out our feature on how to install, update, and use Kodi for more information. On some platforms, such as macOS and Windows, getting started with Kodi is as easy as downloading the installer package. Installing Kodi on an iOS or an RPi device is a bit more involved.

You can sync your media libraries across multiple devices with Kodi, using supported add-ons or a local network. If you want to set up a dedicated media player with your Kodi library in each of your rooms, there’s nothing to stop you from doing so.

Kodi is open-source software developed by the non-profit XBMC Foundation, so many of its features and add-ons are driven by the community. In practice, this means that Kodi sees frequent updates. You can read more about the details of each new release on the official Kodi blog. If you are interested in developing features for Kodi, you can choose to download the absolute latest builds, but most users will want to stick with the stable releases. The latest stable version of Kodi is 18.9 (Leia), but you can download dev builds of version 19 (Matrix).

How to sync your kodi library across multiple devices with mysql

The Trakt Kodi plugin runs in the background of the Kodi media center and automatically scrobbles what you’re watching back to Trakt. The plugin also supports 2 way sync to keep your watched history and collection in sync with Trakt and your other connected devices. You can rate movies and episodes after watching them too.

Trakt is a platform that does many things, but primarily keeps track of TV shows and movies you watch. It integrates with media centers (like Kodi) to enable scrobbling, so everything is automatic. Discover new shows and movies, follow people with similar tastes and voice your opinion by using the Trakt website or one of the many apps built using our API.

Kodi is a free and open source media center for playing videos, music, pictures, games, and more. Kodi runs on Linux, OS X, Windows, iOS, and Android. Learn more and install from Kodi’s website.

Note: We recommend using Kodi 17 to ensure you’re on the latest Trakt plugin.

Scrobbling simply means automatically tracking what you’re watching. Instead of checking in from your phone of the website, this plugin runs in the background and automatically scrobbles back to Trakt while you enjoy watching your media.

How to sync your kodi library across multiple devices with mysql

The following steps will walk you through downloading and install the Trakt plugin from within Kodi.

Note: This guide assumes you are using at least Kodi 17 (Krypton).

  1. Open Kodi
  2. Select the Search icon in the top left

How to sync your kodi library across multiple devices with mysql

3. Select Search add-ons

How to sync your kodi library across multiple devices with mysql

4. Type in Trakt.tv, then press enter or click OK

How to sync your kodi library across multiple devices with mysql

5. Select Script — Trakt

How to sync your kodi library across multiple devices with mysql

6. Select Install

How to sync your kodi library across multiple devices with mysql

7. Wait for the Add-on enabled message in the top right

How to sync your kodi library across multiple devices with mysql

How to sync your kodi library across multiple devices with mysql

After installing the plugin, it should popup with instructions on how to connect your Trakt.tv account. If you don’t see this, choose Configure in the Trakt.tv add-on information screen, then choose How do I authorize…

  1. An 8 digit code will be displayed.

How to sync your kodi library across multiple devices with mysql

2. Go to https://trakt.tv/activate on your laptop, desktop, or mobile device.

If you’re already signed into the Trakt website, you’ll see the screen below asking for the code displayed in Kodi. If you’re not signed into the website, you’ll need to sign in with your Trakt.tv account, or create a new one.

How to sync your kodi library across multiple devices with mysql

3. Click YES to allow Trakt for Kodi to access your Trakt.tv account.

How to sync your kodi library across multiple devices with mysql

4. Return to Kodi after you see the success message.

How to sync your kodi library across multiple devices with mysql

How to sync your kodi library across multiple devices with mysql

After connecting your Trakt account, Kodi will refresh and display the plugin configuration screen.

General

Notice in the top User line, it displays your connected username on the right side. If you want to re-connect to a different Trakt account, choose the How do I authorize… option to go through the connection steps again. You shouldn’t need to adjust anything else on this tab.

How to sync your kodi library across multiple devices with mysql

Exclusions

If you’d like to ignore items from being scrobbled to Trakt, you can do that in this tab. Excluding HTTP sources means any streaming content will be ignore. Exclude specific paths if you’d like ignore local media from being scrobbled. For example, ignore any shows your kids are watching so they don’t scrobble to your Trakt account.

How to sync your kodi library across multiple devices with mysql

Scrobbling

Scrobbling should be on by default for movies and episodes. You can turn on notifications if you’d like to be alerted when data is sent to Trakt. The fallback and offset can probably be left at the defaults.

How to sync your kodi library across multiple devices with mysql

Synchronize

The plugin supports 2 way sync between Kodi and Trakt, manage those settings in this tab. The first option is a good one to turn on, it will run the sync after your Kodi library is done updating. If you’d like to show or hide notifications, you can do that here too.

Syncing ratings is a nice way to populate Kodi with anything you’ve rated on the website or from other Trakt.tv powered apps.

In regards to Movies and TV Episodes, adding to your collection adds items that you have available locally to watch. View your collection on the website so your friends can pick something out to watch the next time they come over.

Be careful with the remove from collection option, this will remove items from your Trakt collection that don’t exist in Kodi anymore. This is generally ok if you only use Kodi, but you might not want to do this if you track your physical media collection or have other media centers connected.

Syncing your watched statuses is a great way to keep your history in sync. Mark something on the Trakt website and Kodi will pick up on the next sync. Mark something in Kodi and Trakt.tv will pick it up.

Playback progress is a really cool setting we recommend turning on. Let’s say you have Kodi installed at home, and Infuse on your iOS device. You can start watching something in Kodi, pause it after 10 minutes, then resume playing at the same position in Infuse on your mobile device! A lot of the other media centers support playback sync too and it’s really handy.

MultCloud solves your syncing needs by allowing you to transfer and sync data between cloud accounts.

How to sync your kodi library across multiple devices with mysql

How many cloud storage services do you use? I am currently using three, all of which sync to my desktop by one means or another. However, syncing between those cloud sources can become a logistical nightmare (on top of which, a recursive headache and add possible redundant syncing, taking up valuable cloud space). When that becomes an issue, what is a multiple cloud service user to do? Fortunately, there’s a service, called MultCloud, that allows you to to transfer and sync data between cloud accounts. MultCloud supports a healthy number of cloud services, including:

Must-read cloud

  • Google Drive
  • Dropbox
  • Google Photos
  • OneDrive
  • Amazon Drive
  • Amazon S3
  • MEGA
  • Box
  • Flickr
  • MediaFire
  • SugarSync
  • MySQL
  • Evernote
  • pCloud
  • And more

MultCloud offers a free account, which allows 50GB of data traffic/month and two sync modes. You can upgrade the free account to either a Monthly ($9.99/month) or Yearly ($119.00/year), each of which offers additional features (visit the comparison matrix here).

Once you’ve signed up for your plan, you can then start working with MultCloud. Let’s see how to add and sync accounts.

Adding an account

Log into your MultCloud account and click on the Cloud Explorer tab (Figure A).

Figure A

How to sync your kodi library across multiple devices with mysql

The MultCloud Cloud Explorer tab, where you can add accounts.

Click the Add Cloud Drives button in the left navigation, select the service you want to add, and click Next. In the resulting window, you’ll be prompted to name the account and then click to add it (Figure B).

Figure B

How to sync your kodi library across multiple devices with mysql

Adding Dropbox to MultCloud.

If you have already signed into that cloud account with the same browser you use for MultCloud, you won’t be prompted for credentials (otherwise, you will be required to log into the account being added). Go back to the Cloud Explorer tab and continue adding cloud accounts until you added all you want to work with.

Transferring and syncing accounts

Once you’ve added all of your necessary cloud accounts, click on the Cloud Transfer tab. In this window (Figure C), select the source and target directories to be used in the transfer.

Figure C

How to sync your kodi library across multiple devices with mysql

Transferring files between cloud accounts.

If you want to sync directories between cloud accounts, click the Cloud Sync tab. In the sync window, click to select your source and destination (Figure D). Give the sync a name and then select the sync type (One-way, Two-way, Other-way).

Figure D

How to sync your kodi library across multiple devices with mysql

Syncing directories between cloud accounts.

Once you’ve configured the sync pair, click the Sync Now button, and the sync pair will be created (but not synced). After the sync pair is created, click the Task Manager button (directly to the left of your MultCloud username. In the resulting window (Figure E), you can click Menu and gain access to options and schedules for the sync pair. Make sure to click the Sync Now button, so the folder pair will actually sync.

Figure E

Our Sync pair task.

Note: Some features found in the Task Manager are only available in the paid version of MultCloud (such as Cloud Sync scheduling).

Simple cloud account sync

If you’ve ever needed to sync cloud accounts to one another (or directories within different cloud accounts), the process has been made incredibly simple with MultCloud. Power cloud users should find this service quite useful. Give it a go and see if it doesn’t solve your multiple cloud account sync needs.

How to sync your kodi library across multiple devices with mysql

Cloud and Everything as a Service Newsletter

This is your go-to resource for XaaS, AWS, Microsoft Azure, Google Cloud Platform, cloud engineering jobs, and cloud security news and tips. Delivered Mondays

I have to implement data synchronization between two big databases which have completely different structures. Basically, I need to grap some data about products in different tables in the first database and re-arrange them for other tables in the second database.

Creating my products on the first time is not very complicated. But I’m looking for a way to update some specific data – not all data – about each product.

Obviously, there are a few issues that make this tricky.

  • I’m not allowed to do anything on the source database apart select queries.
  • On the target database, I can do usual queries (select, update, insert, create) but I can’t modify the existing structure/tables.
  • Target and source db have completely different structures, tables are not the same at all, therefore data really have to be rearranged – comparing tables won’t work.
  • Target database uses a MySQL server – source may be DB2.
  • There are no “updated time” fields anywhere.

So the whole process needs to be done in a single Python (ideally) script.

I think about creating a hash for each product, based on the fields to update in the target database: md5( code + description + supplier + around 10 other fields). A new hash based on the same data will be created on a daily basis from the source database. I will store all hashes in a single table ( item code, current_hash, old_hash ) for performances purpose. Then compare and update the product if the new hash is different from the old one.

There are around 500 000 products so I’m a bit worried about performances.

Is it the good way to go?

4 Answers 4

This is pretty much what I have been doing or a living the past few years, and my gut instinct is that the time to read 500,000 items from the source database and sync in the destination will not take as much time as one might think and the time taken to read the “key” fields, compute the MD5 hash, and cross check with your table to avoid syncing items that haven’t changed won’t end up saving too much time and may even run longer. I’d simply read all and update all. If that results in a runtime that is too long, then I’d compress the runtime by making the ETL muti-threaded, with each thread only operating on a segment of the table but working in parallel.

It would be important to ensure that your destination database has a primary key index or unique index. Otherwise, each of your updates/inserts could lock the entire table. This would be bad if you are taking the multithreaded approach, but important even if you are remaining single-threaded because your job could lock the destination DB table and interfere with the application that rides on top of that DB.

You say the source DB “may be DB2”. When you say “may” it implies that DB is still being designed/planned? DB2 9 or above does have built-in tracking of last update time, and the ability to query and get back only the items that have changed since a point in time. Perhaps this is why the DB was designed to not have a column indicating the last updated time, eg:

The timestamp cutoff for the above query would be the last timestamp your sync ran.

If this is the case, that should solve your problem. But, your solution would end up being tied very tightly to DB2 and in the future they may like to move to another DB platform and expect your sync job to not need to be re-visited. So it would be important to make sure all the right people know that your product will be dependant on remaining on DB2, or if they plan to migrate that migration would include restructuring the DB to have a “last changed timestamp” column, and make whatever changes necessary at the app level to populate that field.