iSpot migration issues.

Please log any migration issues here. Things such as

No pictures
Missing pictures
Funny commenting
Missing content.
Deleted observations loaded:

We will fix these in the mop up!

Please note:
Note yet uploaded - still coming:

  • Projects (those that worked by tags! - NOT those by area: create a place for those)
  • Observation Fields (e.g. Habitats, interactions, project fields).

Please DONT post:

  • Pictures to be rotated.
  • Pictures in the wrong order.
    (But please do fix them if you can).

Please use a heading and then the url:
e.g.

No pictures:
https://www.inaturalist.org/observations/10852817
Missing pictures:
https://www.inaturalist.org/observations/10840705 (1 out of 4, added 2, still 1)
Deleted observations loaded:
https://www.inaturalist.org/observations/10838105 (deleted on iSpot & now iNat duplicate - will delete)

QUICK LINKS TO ISSUES BEING MONITORED:
(no need to report or do anything with these: these are on the mop-up list)

To see observations that migrated without pictures please see:
https://www.inaturalist.org/observations?photos=false&place_id=113055&subview=table&verifiable=any&field:iSpot%20reference%20(2014-2017)

QUICK LINK TO HELP INDENTIFY UNIDENTIFIED OBSERVATIONS:
(note: in the url box add &user_id=XXX to see yours or use the filter box to select a Location that you know, e.g. place=Cape Peninsula or George (to add to url box you will need to know the place code)

No IDs:
https://www.inaturalist.org/observations?iconic_taxa=unknown&photos&place_id=113055&reviewed=any&subview=grid&verifiable=any&without_taxon_id=479641
Weak IDs:
https://www.inaturalist.org/observations?iconic_taxa=unknown&photos&place_id=113055&reviewed=any&subview=grid&taxon_id=48460&verifiable=any&without_taxon_id=479641

SEE YOUR OWN ISSUES
https://www.inaturalist.org/observations?iconic_taxa=unknown&photos&place_id=113055&reviewed=any&subview=grid&verifiable=any&user_id=tonyrebelo

(and substitute your user name in the url box)

https://www.inaturalist.org/observations?iconic_taxa=unknown&photos&place_id=113055&reviewed=any&subview=grid&taxon_id=48460&verifiable=any&user_id=tonyrebelo
(and substitute your user name in the url box)

Posted on 13 April, 2018 16:37 by tonyrebelo tonyrebelo

Comments

Punctuation and diacritic marks
All single quotes (') have been turned into carets (^). Double quotes (") also? This has happened on all migrated text.
All diacritic characters have failed.

Posted by karoopixie over 6 years ago

All diacritic characters have failed. - yes: possibly as data were designated by " and ' and ,.
That we will just have to live with, unless you edit yours.

Unfortunately this destroys links ...

Posted by tonyrebelo over 6 years ago

Missing observation

One of my observations from iSpot has not been migrated: https://www.ispotnature.org/communities/southern-africa/view/observation/409833/disperis-fanniniae-kaapsche-hoop

Posted by caspervanzyl over 6 years ago

Thanks yes: this is an April 2014 observation and the upload is still in 2013. Please check tomorrow (or perhaps even Monday) and see if it is on. Uploads are by date posted, not by user or date observed.

Posted by tonyrebelo over 6 years ago

WHO IS GOING TO POST OBSERVATION 10 000 000
We are almost there. Take a moneyfree bet below!!!

(note these are all data, iNat only celebrates "verifiable" data, so we can have two bites of this pie?
NOW PLACE YOUR BETS!!)

Posted by tonyrebelo over 6 years ago

Tony man, sorry. I only now discovered that the complaints desk has been centralised here. Since last night I've @ed you a hundred times inside observations. I hope you find them all.
=RiaaN=

Posted by beetledude over 6 years ago


WHO IS GOING TO POST OBSERVATION 10 000 000



Two safe bets are tonyrebelo and nicky.

Posted by beetledude over 6 years ago

How do you know which is obs number 10mil? I've been asking ppl to clean up the "unknowns" here: https://www.inaturalist.org/observations/10819353, there are 3000 with 1/3 of the way through, so they'll be around 10k to clean up in the end.

Posted by vynbos over 6 years ago

@beetledude - with over 530 comments, the system failed at about 300, so I have not seen them all.
Please though: only put migration issues for my attention here, not iSpot, iNat or ID & agreement issues (incl. hemihomonyms and fuzzy mismatches: please address those to the broader community or flag me with the @ on the actual observation itself.

@vynbos - please keep up the stirling job. Will ask others to help out too.

Posted by tonyrebelo over 6 years ago

Observation 10000000 went by so fast I have no idea who it was.

Posted by tonyrebelo over 6 years ago
Posted by magdastlucia over 6 years ago
Posted by magdastlucia over 6 years ago
Posted by vynbos over 6 years ago
Posted by magdastlucia over 6 years ago
Posted by magdastlucia over 6 years ago
Posted by magdastlucia over 6 years ago

All ispot migrant obs without images
19 april 7am: 7484 obs
If only one could extract a CSV from a filtered view.
Update: actually one can, so @tonyrebelo can DL'd the list when the migration is complete. So is it really worth Magda's time putting up every one of her missing pic ob here?

Posted by vynbos over 6 years ago

Extracting data to csv. You can!! You need to attend the course!!

No photos for migration
https://www.inaturalist.org/observations?photos=false&place_id=113055&subview=table&verifiable=any&field:iSpot%20reference%20(2014-2017)
Not photos for all southern Africa
https://www.inaturalist.org/observations?photos=false&place_id=113055&subview=table&verifiable=any

For any view of observations (i.e. in "Observations" tab of the top iNat menu bar)
click on the grey "Filter" box
check the filters are OK
click Download (bottom right hand corner)
Select the fields that you require (in this case, please include "iSpot reference (2014-2017):"
download

Simple as that!!
(beware hidden coordinates are hidden! - include Geoprivacy field in your fields to check)

Posted by tonyrebelo over 6 years ago

I just said I found out how!
Anyway, back to my question, is it really worth Magda's time posting all her missing pic obs?

Posted by vynbos over 6 years ago

Yes, it will allow her to check hers afterwards, and make sure none were missed. An independent check.
But we will correct it from the automated list.

Posted by tonyrebelo over 6 years ago

Hi Tony, just to be clear that I understand you correctly...
I see that 639 (so far) of my observations that have been migrated so far have come through without photos .
I saw this by using the link you provided above and added my username
Would these all be on the mop-up list that you refer to?

Posted by flower_girl over 6 years ago

Yes: these are all for mop-up next week.

Posted by tonyrebelo over 6 years ago

@tonyrebelo I assume our iSpot obs that weren't from s.Africa (like mine from the Alps)will not be migrated?

Posted by vynbos over 6 years ago

Correct, unless you especially request them ....
But they will not have all their data: only what you can see.

Posted by tonyrebelo over 6 years ago

Hi Tony, is it possible that some of my observatiosn on iSpot have not migrated at all?
I have just been looking for an observation from 2 years ago (Pelargonium denticulatum (19/7/2016) ) and I can't seem to find it. It seems that only 3 observations from this date have migrated and I must have posted at least 20 .

Posted by flower_girl over 6 years ago

@flower_girl is it this one? https://www.inaturalist.org/observations/11261648 perhaps you still have the filter on that excludes obs without photos. You had 945 to be moved and 945 were moved. You had 3 obs on iSpot for the 19-7-2016 and 3 obs are on iNat for that date.

Posted by vynbos over 6 years ago

@vynbos, no that's not it. That's funny because I posted at least 20 observations on iSpot for that date, including one of P.denticulatum.
Oh well, perhaps I will repost the missing ones...
By the way, when might the missing pics be coming through?
Thanks for all that you're doing!

Update: I (sheepishly) now remember that I deleted those observations. Have reposted them.

Posted by flower_girl over 6 years ago

https://www.inaturalist.org/observations/11259662
Not sure what happening here. My originals are correct, but this seems to be zoomed in and clipped.

Posted by shauns over 6 years ago

see https://www.ispotnature.org/communities/southern-africa/view/observation/678750/knysna-lourie-hogsback - I can only copy what is on iSpot, and that was what was copied. Sorry: but this is an iSpot problem and nothing I can do about it. Apologies.
Please reload your originals.

Posted by tonyrebelo over 6 years ago

Thanks. Will do

Posted by shauns over 6 years ago

Hi Tony, any news on all the missing photos on my observations? Thanks.

Posted by flower_girl over 6 years ago

You are too impatient. We have not yet finished scraping this batch of photos off iSpot yet (yours are already off), so we wont begin until this is done. Please give us a moment.

Posted by tonyrebelo over 6 years ago

Thanks Tony - no it was just a gentle enquiry as to the progress. I can only imagine what a huge task it must be and I am very grateful!
:-)

Posted by flower_girl over 6 years ago

Hi Tony, I am also still missing the photos of all my iSpot observations. I know that I am impatient, but I want to delete my iSpot account for some reasons a.s.a.p. :-) Br Detlef

Posted by detlef over 6 years ago

Will try and find out what is happening. Give us a few days please. But dont delete anything after 23 June yet please - I need a few more weeks for those.
Ta
Tony

Posted by tonyrebelo over 6 years ago

Still no sign of the migration. Scott (last night) has handed the scripts and data to Patrick who has to queue and run them. This is tedious.

Still to come: pictures for 12,000 observations and thousands of Projects and Observation Fields.

And then the second and last batch of data. But that is at least a month away.

Posted by tonyrebelo over 6 years ago

And then:
Insert southern African Common Names
Tweak iSpot for the southern African community (waiting on SANBI to finalize the MOU with iNat).

Other issues we are just tackling ourselves.
Tribes and non-GBIF taxon ranks (e.g. subfamily, tribe, subtribe) - esp. crucial in Flies and Beetles.
Subgeneric rankings.
Mismatches between iNaturalist dictionary and BODATSA and ZODATSA (these were just imported in parallel, and remain to be sorted when users point them out). Daisies, Moths and Beetles are possibly the worst affected. If you discover things like Osteospermum monilferum AND Chrysanthemoides monilifera - both active species, then please notify Riaan (inverts) or me (plants). We cannot promise immediate action, but we will schedule it.

Subgenera
We hope to add a lower classification to all plant genera with more than 100 species in the Cape. (other regions, please find your own champion). These are (= done):
Erica
, Aspalathus, Restio*, Agathosma, Pelargonium, Phylica, Cliffortia*, Oxalis, Moraea, Senecio, Muraltia, Gladiolus*
We have also done Protea*, Leucadendron*, Leucospermum*.

A condition of our doing this is that genera (or families, tribes, etc.) have to be done worldwide, making the task a little harder for taxa extending into poorly known tropical African (or indeed worldwide) regions.

Posted by tonyrebelo over 6 years ago

Migration has recommenced. Pictures first. 12000 and getting less. Then Observation Fields, then Projects.

Posted by tonyrebelo over 6 years ago

Phase II is completed for pictures.

The breakdown is as follows:

Pictures did not download for 612 Observations (307 species 82 identifiers) and 27 observers.
Ignoring Peter who did his own thing
1 peterslingsby 415 - complex reasons
2 tonyrebelo 91 - complex reasons

the breakdown is:
3 shauns 33 - all these are deleted on iSpot:
4 timvn 19 - these are all "missing" on iNat
5 charlstrydom 12 - complex: some missing, 8 failed to migrate pictures
6 beneke 5 - these are all "missing" on iNat
7 jean_s 5 - these are all "missing" on iNat
8 mr_fab 4 - failed to migrate pictures
9 rov_ryan 4 - these are all "missing" on iNat
10 soulsurfer 4 - failed to migrate pictures
11 eugenemarinus 3 - these are all "missing" on iNat
12 anthonywalton 2 - failed to migrate pictures

The following are all all "missing" on iNat (or * = failed to migrate).
13 alison_young 1 *
14 dlmaitre 1

15 kirstenpacker9 1

16 littlepaws 1 *
17 lizziepop 1 *
18 magdastlucia 1 *
19 michdagforcioli 1

20 ntuthukomabuya 1 *
21 patrick_fraser 1

22 pbsouthwood 1 *
23 rosepalmer 1

24 ruedi_siegenthaler 1

25 sandraf 1

26 saphotographs 1

This is a legitimate pictureless observation:
27 tuli 1 0

The iNat team are trying to figure out why the missing observations are not missing on the summary of observations present without any photographs. Then we will be able to evaluate if they are deleted, missing pictures on iSpot or some other issue. So please Hang Ten.
Unfortunately, all these require user action, as we dont have access to your accounts or data: they are yours alone and only you can fix them from now on.

Please note instructions below.

Posted by tonyrebelo over 6 years ago

INSTRUCTIONS!!!

Instructions: if your observations are listed as "missing" please wait further instructions.

Attention - see below: esp. @peterslingsby @charlstrydom @mr_fab @soulsurfer @anthonywalton

Please go through your observations:
https://www.inaturalist.org/observations?photos=false&place_id=113055&subview=table&verifiable=any&user_id=YOUR_USER_ID - replace "your_user_id" with your iNaturalist user_ID

If there are any observations listed

open your observations.
(if they are missing on iNat and you cannot open them - you get the "mole", then please await further instructions. No need to do anything more right now)
click on the iSpot link to see what the observation looks like on iSpot

*** If it is a missing picture(s) on iNat and you have the pictures.

upload your photos by going to your album directory, selecting the pictures and dragging them onto the add-picture icon. (You can go to the next observation and do the same or do anything else: iNat will manage the upload provided you leave the tab in your web-browser open in the background.)

*** If it is a missing picture(s) iNat and you DONT have the pictures.

on iSpot select the picture on the picture carousel,
5b. click on full size
5c. right click and "save image as" and give a directory on your computer (or your desktop)
5d repeat for all pictures until they are all in your directory
5e: do item 4 above.

*** If it is a missing picture(s) iNat and iSpot and you DONT have the pictures.

Decide what to do with the obervation.
6b: delete it if you dont want it, or it was a mistake, or you dont know what on earth it is about.
6c: leave it as a pictureless observation, but then please add a note that you dont have any pictures.

*** If it is a missing observation on iSpot (iSpot says NOT FOUND), then
it is either because you deleted it after 23 June 2017, or because of an iSpot bug.

Decide what to do with the obervation.
7b: delete it if you dont want it, or it was a mistake, or you dont know what on earth it is about.
7c: add pictures (do item 4 above) if you have access to your pictures.
7d: leave it as a pictureless observation, but then please add a note that you dont have any pictures.

If you dont know what to do, then leave me (@tonyrebelo) a message on your observation and explain the issue.

That almost concludes the migration of the pictures. We just need to find out why iNat is giving us the mole for about a dozen observations.

Posted by tonyrebelo over 6 years ago

< h3>BUT MY OBSERVATION WAS NOT MIGRATED!!!

We can still migrate more data if you missed out.
DEADLINE: 22 June 2018.
Note that it will be incomplete compared to what has already been migrated: we do not have as much data, and the new iSpot site hides more data than it used to. But all the essentials will be done.

*** my data is was posted before 23 June 2017.
It should have been posted.

Did you give permission? (if not do so, it will be included in Batch II)
Is it from southern Africa? (if not see below: Extra data)

*** my data is from 23 June to 25 September 2017.
These will be done in Batch II. Please Hang 10. We are busy extracting these data from iSpot. If you gave permission to SANBI for Batch I, any data for the period June to Septemberr will be migrated in the next Batch. Details to follow in July.

Is it from southern Africa? (if not see below: Extra data)

*** Data is after 25 September.
We will need your special permission. We will also need a list of each observation that you want imported. Only listed observations accompanying the signed permission will be processed. The list can be in excel, word or text file.

The list to look like this:

https://www.ispotnature.org/communities/southern-africa/view/observation/100/grazed-to-gone
https://www.ispotnature.org/communities/southern-africa/view/observation/1062/palmleaf-sorrel
If any data are obscured, or are of sensitive species, then we cannot import it. Please provide a list of these observations with the details of the locality (latitude, longitude, locality, locality details), otherwise they will not be included.

***Extra data

"I have data from outside of southern Africa that I wish to have migrated. It is from various periods."

We will need your special permission. We will also need a list of each observation that you want imported. Only listed observations accompanying the signed permission will be processed. The list can be in excel, word or text file.

The list to look like this:
https://www.ispotnature.org/communities/global/view/observation/111/condors-arise
https://www.ispotnature.org/communities/chile/view/observation/1234/palmtree-sorrel
If any data are obscured, or are of sensitive species, then we cannot import it. Please provide a list of these observations with the details of the locality (latitude, longitude, locality, locality details), otherwise they will not be included.

Anything else!
Leave a comment below.

Posted by tonyrebelo over 6 years ago

Incomplete picture uploads: SUMMARY

https://www.inaturalist.org/observations/11218794 - 1 of 2 uploaded - manually fixed.
https://www.inaturalist.org/observations/10983298 - 3 of 10 not loaded - added.
https://www.inaturalist.org/observations/11247477 1 of 3 uploaded - added

Posted by tonyrebelo over 6 years ago

"The list can be in excel, word or text file. The list to look like this: " then you give two links, one is to an obs, the other is a dead link. So I'm not sure what you mean, Did you mean to say "the URLs to look like this"? Did the spreadsheet with the URL of all my European obs sent on 28th of April comply with what you need?

Posted by vynbos over 6 years ago
Posted by magdastlucia over 6 years ago

@vynbos
(all examples are duds - the actual numbers are nominal)
This is what I have at present (format perfect):

Extras:
Nigel Forshaw 111
vynbos 80
colin 121

New Permissions:
Joanette Le Cornu 692
Chris Walker 256 (Got pictures Originals lost due to wrong iNat user)
Brittany 184

lucanus 175
Martli Slabber 148

Christeen 38

pieterws 19
Costas Zachariades 0

Posted by tonyrebelo over 6 years ago

Updated missing pictures list.
The iNat team have resolved the issue of the mising observations: this is the list of observations missing pictures.

https://www.inaturalist.org/observations?subview=table&verifiable=any&field:iSpot%20reference%20(2014-2017)&photos=false&view=observers

At present this is:
415 peterslingsby
91 tonyrebelo
33 shauns
8 charlstrydom
4 jean_s, mr_fab (2)
2 anthonywalton

1 alison_young, littlepaws, lizziepop, ntuthukomabuya, pbsouthwood (5)

Posted by tonyrebelo over 6 years ago

RE: uploading of habits, is this still on the card? is it possible for the process to automatically mark the obs as 'not wild' if the habitat is gardens, zoos etc?

Posted by vynbos over 6 years ago

The iNat team have been promising this for many weeks. Apparently it is all tested and OK. But they have not yet loaded it.

Marking not wild. No - sorry: too late now. Possibly if it had been requested when I did the programming in February.
But I am against it.

Firstly, just the term "research grade": that GBIF data are research grade and nothing else pisses me off extremely. Good data gets made casual because GBIF does not want non-wild data. So all urban and garden plants are not research grade, despite being good scientific data.
Secondly, flagging it "not wild" removes it from the "needs ID" queue, when as far as I am concerned, I often post planted stuff (esp in urban parklands) because I would like a name, and as "casual" I am unlikely to get a name.

Posted by tonyrebelo over 6 years ago


@tonyrebelo, look at the current distribution map of #Euryops pectinatus pectinatus. This obs is causing the problem: https://www.inaturalist.org/observations/11152861
So how do you distinguish between good garden data and bad garden data?

Posted by vynbos over 6 years ago

By:

Not confusing research grade with wild. The two are not the same. So the progression: casual (no ID) > needs an ID > research grade (has an ID) is not the same as wild > invasive > planted! This is "suitability of data for GBIF", which does not distinguish wild from not-wild.

By allowing us to switch on and off "not wild" data on the maps, or displaying them with different symbols. You could make exactly the same argument for "natural distribution" and "invasive distribution".

But what do you mean by good versus bad data? Good data has a locality, date and potential identification. Bad data has a wrong locality, wrong date or wrong identification. Two bring in variables like good "garden' or good "natural" is going into different realms of users, needs, agendas which are all variable and sometimes complex.

Please take this issue further to the forums: not here.

Posted by tonyrebelo over 6 years ago

Mismigration: Biocontrol Project
This should not be on the following:
https://www.inaturalist.org/observations/12358976

Posted by tonyrebelo over 6 years ago

Is it still possible to give permission for my Milnerton Racecourse data to be transferred from ispot to inaturalist or am I almost a year too late?

Posted by landiretief about 6 years ago

I guess om way to late to have my profile moved over to inaturalist from ispot now right?

Posted by bugmanslife about 5 years ago

Waaaay too late. What was your user name ...

Posted by tonyrebelo about 5 years ago

Then it's no worry! But my username was BugMan

Posted by bugmanslife about 5 years ago

I might be able to sneak you in amongst the mopping up (about 200 observations did not upload for some reason). Please send me an email giving me permission to copy your data across.

Posted by tonyrebelo about 5 years ago

Tony if you can sneak me in too for Milnerton Racecourse sightings then great, my user name was Landi

Posted by landiretief about 5 years ago

Add a Comment

Sign In or Sign Up to add comments