Need help sorting out how to use 23? Or have you discovered one of those nasty bugs?

serious upload trouble

nachbarnebenan plus   April 10, 2012, 10:43 PM

For about three or four days now uploads tend to fail a lot. Batch uploading became almost impossible as images from the batch randomly fail and you basically have to upload them one-by-one, checking after each if they got through, to keep them in order.
About a third of the images take two tries to succeed and a handful need three or even four tries.

The error message is always the non-descriptive
"These files couldn't be uploaded (possibly because the file type isn't supported):"

No idea what's wrong, but maybe the mimetype detection is on strike? Steffen, could you take a peek at the logs to see what's going on?

 
Bret Pulkka-Stone   April 11, 2012, 06:42 AM

exactly the same here, I figured my .jpgs were corrupt. Small ones go through, big ones don't.

- Bret

 
Steffen Tiedemann Christensen Team 23   April 11, 2012, 06:44 AM

Can you throw me an example of ones not going through at steffen@23company.com?

 
Bret Pulkka-Stone   April 11, 2012, 11:22 AM

I will retry a couple of the big ones... three went through, one didn't.

You have mail :-)

- Bret

 
tridymite plus   April 11, 2012, 01:49 PM

Happend to me too, thought it was my internet (it fails a lot lately)

 
nachbarnebenan plus   April 11, 2012, 10:58 PM

It didn't get much better yet, but at least no file failed more than twice this time. I've uploaded a screenshot as to how it looks. 7 images out of 25 failed and two of these only made it in the third attempt.
But in the end, all of them were accepted.

Steffen, if you still need examples I can send these to you but they're quite large.

 
nachbarnebenan plus   April 12, 2012, 11:04 PM

It's gotten better and worse at the same time. I had to re-upload two images five times until they were accepted. But the overall number of files that are rejected has gone done.

 
nachbarnebenan plus   April 14, 2012, 12:11 AM

Ok, there have been improvements, even though it now takes quite long for the thumbnails to be generated.
My guess is, there was a time limit for the images to be processed after upload and that failed either due to memory/cpu constrains or a race condition. It now seems as if this has been serialized and even though slow, works much better.
There are still some bugs, but only one image was rejected this time and I had to delete and re-upload a few photos but I think this was due to the servers not being available for some minutes.
So, over all this appears almost fixed.

 
nachbarnebenan plus   April 19, 2012, 05:34 AM

It's now the third day everything seems to be fine again. Upload speed is back to normal, images are no longer rejected and all thumbnails are created in time.
Thanks for fixing it!

 
Steffen Tiedemann Christensen Team 23   April 20, 2012, 07:17 PM

The annoying thing is that we did nothing from our end. I've been running profiling for the past week or so to figure out the reasons -- and haven't come up with a good explanation. There you go though, a win for the good side ;-)

 
nachbarnebenan plus   April 20, 2012, 07:49 PM

That's worrying. Is there is way to get more information about what happened when something goes wrong (other than sending you a mail in the middle of the night to look at the servers)?
Something like a "dump debug log" link that gets visible when a file got rejected and which will include the current status of the upload processes so you can later have a look at them.
Or the reverse, for some time log all the upload/processing activity (probably to a separate machine) and have a "there was a problem" as well as a "everything went fine" link users would get in their option screen or somewhere else easily reachable but separate from the upload page. Let's say about an hour or so after the last completed upload (independent if it was successful or not) the log is deleted, as well if the user clicks the all-is-well link. Otherwise, if the user clicks the complain link you get the log. Maybe even keep a few randomly selected successful logs around to run comparisons.
I'm not sure about this — but I want the problem gone. This morning after 17 images the upload stalled (with the last image thumbnail being generated incorrectly) and when I finally restarted it the last 8 files took more than 5 to 6 minutes each to upload. It's really difficult to help you track this down as we can't give you much more information other than that something went wrong.

 
Steffen Tiedemann Christensen Team 23   April 20, 2012, 07:58 PM

I agree. For now, I have spent time upping the logging behind the scenes of uploads working and failing -- and have plans to add more during the weekend. There has been small tweaks as a part of this, which could also account for the improvement. If this happens again, throw a message in this thread. In the meantime, monitoring is going on.

 
Bret Pulkka-Stone   April 20, 2012, 08:21 PM

What I can do is upload a bunch of preprocessed JPGs and see what happens? The connection from here is pretty good in the mean time. You would like UTC timestamps?

I need to process a massive set from last weekend, so I'll start that now and see what happens when I upload the final results. Should be around 50-60 ca.10MB JPGs, so if there's going to be a problem then I should hit it. Failing that, I can upload a complete directory's worth of JPGs to prove it... but I'd rather not :)

- Bret

 
Bret Pulkka-Stone   April 20, 2012, 09:58 PM

currently uploading 40 or so JPGs; the receiving server is slow (.17.58) but it all appears to be going through ?! thumbnails are fine. I'll retry a whole bunch of the stuff that was problematic before tomorrow and see what happens.
It seems to take 30s for the preview to be generated and the transition to the next file also takes a perceptible amount of time...

- Bret

 
nachbarnebenan plus   April 20, 2012, 10:31 PM

I can confirm this — 25 image, each about 50meg just ran through without a hitch. Thumbnails generation is also quite fast.
I guess the servers behave as long as Steffen is watching…

 
Steffen Tiedemann Christensen Team 23   April 21, 2012, 06:27 AM

The better behave! I love the energy of having you guys throw photos at us to see the results ;-)

 
nachbarnebenan plus   April 30, 2012, 12:12 PM

At the moment, upload is very slow again. But files do not fail (up to now).

 
Steffen Tiedemann Christensen Team 23   April 30, 2012, 12:25 PM

Okay, we're throwing hardware at the problem now; we're set to launch a new server center for 23 some time next week. So exciting thing coming soon ;-)

Steffen

 
nachbarnebenan plus   April 30, 2012, 12:57 PM

> Okay, we're throwing hardware at the problem now;

Urgs. Hardware? Then it must be really bad…
Or are there so many 23 users now (which would be good)?

I'll upload less images per day next month anyway, so it should still be ok — I hope.


ps: Please also try to do something about the captchas (like only require them for the first two or three comments per user and thread), they're getting ridiculously unsolvable.

 
Steffen Tiedemann Christensen Team 23   April 30, 2012, 01:03 PM

Yup, weird recaptcha is getting to the hard books these days.

The real reason for the hardware overhaul is 23 Video, and we're quite excited about it. Would have loved if 23 was causing the problem ;-)

 
To participate in this conversation, you'll need to join the group




About 23

About 23
What is 23 and who's behind the service?
Just In
Discover the world from a different angle.
Here's a crop of the latest photos from the around the world.
Search
Search photos from users using 23
Help / Discussion
Get help or share your ideas to make 23 better
23 Blog / 23 on Twitter
Messages and observations from Team 23
Terms of use
What can 23 be used for and what isn't allowed
More services from 23
We also help people use photo sharing in their professional lives
RSS Feed
Subscribe to these photos in an RSS reader
  • Basque (ES)
  • Bulgarian (BG)
  • Chinese (CN)
  • Chinese (TW)
  • Danish (DK)
  • Dutch (NL)
  • English (US)
  • French (FR)
  • Galician (ES)
  • German (DE)
  • Italian (IT)
  • Norwegian (NO)
  • Polish (PL)
  • Portuguese (PT)
  • Russian (RU)
  • Spanish (ES)
  • Swedish (SE)

Popular photos right now

See also