MTSv3 auto-download - how to get history before last 45 days on it?

MTSv3 auto-downloaded on my system and I just saw and have started using it. It synced the last 45 days automatically when I opened the tracker but it's left all the data before that empty. How do I get the data before the last 45 days in there?

@Kadauchi

 

Follow the instructions for this: Export HIT Tracker in multiple files instead of one large file (github.com) and then import these files.

All the data is there, just not being accessed correctly.

 

Thank you! But I can't get it to work. v3 automatically downloaded on my system so there is no v2 anymore (the instructions on the github page say to use the script on v2). I tried on v3 anyway but it doesn't work. I get this message:

Also, I am the one who had trouble having correct pending hits and pending earnings counts; some random days in the past had bugged or something and remained always as pending, making the pending hits and pending earnings both always much larger than they should be. I could not export all my hits at once to re-import them; any export attempt like that would freeze. Also, I couldn't do any requester searches for all days and things like that on tracker; any attempt would also freeze, probably on the same bugged days. Long ago you'd told me I might need to wait for the next version which I suppose v3 is. I didn't know v3 was coming so a few weeks ago I asked again and you gave me this same script as above to use in v2; I tried it then and it worked until it got to one of the days that were bugged and then froze and wouldn't go farther.

@Kadauchi

 

Everything from MTSv2 still exists in the browser, it gets converted and put in a new database for MTSv3 and then the old database is left in case issues arise.

Did you uninstall MTS at any point?

 

@Kadauchi
I updated yesterday. Nothing works now.
Note: Nothing got converted in my browser (Chrome Version 87.0.4280.141), and I never uninstalled MTS at any point.

So this is the 2nd day that I can't access HiTs from MTS. I asked yesterday in a thread and have received no reply. I have asked for instructions on how to use/setup new version so that I can use it. A pdf would have been helpful. Also have sent emai about this. Radio silence so far. I realize you all are busy, but not even an auto-reply so that I know someone received and will get back to me?

I am a paying subscriber and am now wondering why I am when I can't use MTS to make $$.

 

I'm going to add a MTSv2 tracker export into v3. This will be available in 3.2.2 (3.2.1 is already done and pending review in the extension stores).

 

duplicate.

 

Nope I've never uninstalled mts. When I quit my chrome browser and reopened it had updated to v3 automatically.

I have had issues with updating hits and syncing even in v2 too though, as mentioned. I know that it's particular days that are/were bugged because when you gave me the alternate script a few weeks ago that exports in smaller batches (the one you also posted above), it worked until a certain batch but then would always freeze at that batch. Also near the end of 2020 another day bugged immediately and I know exactly what day it was (it was a day in early October) because before it had fallen off the 45 day mts-sync threshold I could see on mturk that hits from that day had approved, but on mts they never showed as approved and I couldn't even sync my last 45 days because it would always freeze on that day. So, the days before that particular day would sync but then the sync would freeze so not the days after. Then when that bugged day finally fell off the 45-day mark, all the last 45 days would again sync when I tried. That day from early October 2020 and others from way before like it are somehow bugged and causing problems. I don't know, however, if that's related to no history at all showing up in v3 past the last 45 days.

@Kadauchi

 

Don't mean to bud in to much on this convo, but I was wondering how old is your computer and how much ram do you have?

I ask, because I remember someone from a few years ago had something similar to this, and they had only 4gb of ram and a oldish computer.

 

Thanks; my computer is from about 2015. It has 16GB ram. It's a Mac too if that matters.

 

This is for if/when 3.2.2 fails to export/access your v2 database.

Would you be comfortable using something like Chrome Remote Desktop if needed?

If memory is an issue (16GB 100% shouldn't) or if its related to the DB just being too big then I can make a destructive exporter that will export out in chunks and delete after it has been exported so when it freezes and gets reran it wont have to parse over already exported data. This is the last thing I want to do though.

 

Okay, crossing my fingers 3.2.2 does it!

Thanks for the offer; I've never used Chrome Remote Desktop before but I think would be willing to if needed.

The DB is very big so that could be it; I still would be curious why it picks only a very few certain random days to bug on and leaves them that way. I wonder if even the destructive exporter would work. It seems like it would freeze on a bugged day thus leaving the chunk with that bugged day unexported. So when it restarts it would start back at the chunk with the bug and freeze right away, if I'm thinking through what you're saying correctly (or delete the entire chunk with the bug, including good unbugged hits).

Now that I think on it, I've thought of a similar manual way that might work to possibly export all the hits minus those bugged days wherever they are. It might be cumbersome but if it works then doable. When I still had v2 and tried the script you gave me a few weeks ago (the same one you posted above), it did work until a chunk with a bug. I'm no expert on these things but I looked at what you'd written and noticed

I changed it to

and it ran and froze on a chunk within the range of the same original 100000, so I was able to narrow down the first bugged day to a range of ten thousand hits. Every time I tried it froze on the same chunk.

What I'm thinking is that I could keep lowering the count on maxHitsPerFile on the chunk that freezes (to 1000, 100, 10, 1) until I find the exact first hit that freezes it. Then I'd have everything exported up until the first bugged day.

After that, I'm not sure if it's possible but if it is, some instruction in the script could be added to start from the hit number after the bugged hit. Like, if the first bugged hit is say 17362 then the script could say to start the export on 17363. I don't know if only one hit in a day is bugged or an entire day's hits, so if it freezes right away I'd have to keep adding 1 to the number until it works again for the rest of that small chunk.

Then, I could just keep doing all that same process over again until I find the next bugged day and so on until everything is exported except the bugged days, and then reimport without the bugged days and all should (?) finally be good then. Right now I can't try it because I'd have to go back to v2 which I don't know how to do, or wait until v3 has all my history. I see the similarities in this to what you were talking about with the destructive exporter; it would sort of be like a manual version of that with manually overpassing bugged hits if such a thing is possible, if the destructive exporter won't work, and if it comes to all that.

What do you think, @Kadauchi ?

 

It kind of sounds like its not dropping the HITs it has already exported out of the memory so it just keeps growing until it fails. That'd explain why it always fails around the same count. I don't think its a corrupted HIT since that shouldn't even be possible but you can check this by running the export script again but with the console log open (ctrl+shit+j). If an error pops up when it fails its something with the HIT, if it just fails its most likely memory or some similar issue with it not forgetting the already exported HITs.

Run the export script on the options page instead of HIT Tracker in v3.

I'll try to make a change to the export script today which I'll see if I can make it start from a specific index. I'll ping you when that is ready to try.

 

Thanks for the quick reply. The script does run on the options page. With 100,000, it always exports the first 100,000 and then freezes on the second (so only one file is exported, the "1 to 100000"). When I change it to 10,000, it always exports up to the file of "160001 to 170000" and then freezes. When I change it to 1,000, it always exports up to the file of "175001 to 176000" and then freezes. When I change it to 100, it always exports up to the file of "175201 to 175300" and then freezes. I'm not going to try 10 or 1 just yet because that would create thousands or ten thousands of files, but it looks like the first day or hit that is causing it to freeze is between 175301 and 175400. These are the same numbers it stopped at a few weeks ago too (then I only tried the original 100000 and 10000).

No error message that I can see is being displayed on the console, so that's good news. It just says

I can open that and it gives me a lot more info but from what I looked at I didn't see any error messages. When I say freezes, I mean it just stops downloading any more files or doing anything, so in essense freezes or quits (it's similar to when I would try to do a full export on v2, but since a pop up on v2 full export showed higher and higher numbers as it works through the hits, I knew it had frozen because the changing numbers would freeze).

I just thought to try the normal export on the v3 hit tracker. It did download a file of "mts backup tasks", but it did so extremely quickly, so the file may just contain the last 45 days info that the v3 hit tracker screen currently does. Yes I'm pretty sure that's the case because the 1 to 100000 download file from the script you gave me is 150 MB but the normal export button on the v3 hit tracker downloaded a file that's only 2 MB.

@Kadauchi

 

I am not a programmer, but have been turking since 2015. MTS updated automatically and I didn't know I had to have a tracker backup for the update. My last backup is from October(better than nothing...) I tried to import the file- chose the V2 button since I get an error on V3. Got the "61853 MTSv2 tasks have been imported!" message.

I can't search anything. I have hits there if I just press the day/week/month buttons, but otherwise it just spins. I have never uninstalled MTS. I am on a 1 yr-old Chromebook Chrome version 87.0.4280.109.

Is searching just not a thing yet?

 

There have been multiple reports with searching not working correctly. I have a fix coming in 3.2.2 but if that doesn't fix it for you let me know and I'll investigate further.

 

It looks like mts is now up to 3.2.2 but it's still not showing any other hits from before on my tracker history. Actually, I looked and the tracker history doesn't even go back 45 days, but only back until January 3rd.

I tried exporting and it's still doing the same thing. The mts3 exporter only exports the last few weeks, while the mts2 exporter does the "1 to 10000" just like the script you gave me and freezes same as the script.

@Kadauchi

 

I was able to make an export file with the new version - although it says there are 74k+ but imports 64k. I tried importing and IDK maybe it has - I can't search anything. It just spins and spins.

I can see what's been done this day/month/week/year, but search doesn't do anything.

 

I imported to Edge this morning to see if it would work. Nope. I can only find things for the last 45 days. Anything before that comes up blank, although it told me my 164,255kb file imported successfully. I exported the v3 (2 files), but got an error when I tried to import them in v3, so had to import v2. So then I imported my old backup to v2 and still nothing older than 45 days.

 

I just noticed that I think 3.2.5 finally fixed the pending hits/earnings issue! After years now the extra stuck pending hits are gone! Woohoo! It is SO great to be able to see exactly what's pending again. Thanks @Kadauchi

Now the only thing is to get my old history pre-2021 in the data again, which you've already given me directions to do so when I can I'm going to try it.