can't download dropbox public links

I've never had a problem downloading dropbox public links with lua before, until recently. The url I'm trying to download is https://dl.dropboxusercontent.com/u/2138237/version.txt

I'm not having any problems downloading stuff from other websites. Whenever I try downloading a dropbox public link, it returns a http 302 error for some reason. I've also tried modifying the link, following advice in dropbox's help centre like adding ?dl=1 to the end of the url, but I've not had any luck.

Is anyone able to download any dropbox public links with lua? I'm inclined to think that this is something that is caused by a dropbox change, as I've not made any changes to my code for a long time.

LuaSocket can't do HTTPS and instead just silently shunts you to HTTP, which will not work on the modern secure web.
You will have to also wrap your connection with LuaSec.



I ran into the same thing when I started using github, which also does not allow HTTP connections.

I now have a wrapper for doing asynchronous HTTPS using LuaSec, LuaSocket, and lua-llthreads.

Relevant files in my repo are (I think this is all of them):
https://github.com/fiendish/aardwolfclientpackage/blob/MUSHclient/MUSHclient/lua/async.lua
https://github.com/fiendish/aardwolfclientpackage/blob/MUSHclient/MUSHclient/lua/ssl/https.lua
https://github.com/fiendish/aardwolfclientpackage/blob/MUSHclient/MUSHclient/lua/ssl.lua
https://github.com/fiendish/aardwolfclientpackage/blob/development/MUSHclient/llthreads.dll

---

[EDIT] I've since switched from LuaSec to Lua-openssl because luasec was not thread-safe and the lua-openssl creator actually pays attention to pull requests and comments.
https://github.com/zhaozg/lua-openssl

The complete set of files related to sockets, ssl, and threading should be:

MUSHclient/socket/*
MUSHclient/lua/ssl/*
MUSHclient/mime/*
MUSHclient/lua/async.lua (for asynchronous background sockets)
MUSHclient/lua/ssl.lua
MUSHclient/lua/socket.lua
MUSHclient/libeay32.dll
MUSHclient/llthreads.dll (for asynchronous background sockets)
MUSHclient/openssl.dll
MUSHclient/ssleay32.dll

You might also need to replace your lua dll with mine (there are two), since I use LuaJIT instead of PUC-Rio Lua, and then also include the msvcr100 dll if you don't have it in your system32 directory.

As Nick mentions below, Dependency Walker is a great tool for determining what's still missing.

---

Then...


Then somewhere where you want to initiate the background request...


Then at some later point when you expect the request to be complete...


The async stuff isn't necessary, it just prevents long pauses on slow server response ( you can test with http://fake-response.appspot.com/?sleep=5 ) because the version of LuaSec I'm using didn't seem to support it natively.

error loading module 'llthreads' from file '.\llthreads.dll':
The specified procedure could not be found.

Edit: I've also downloaded the aardwolf package update checker and installed it into my mushclient installation. I think its also unable to load the dll as I get the following error:

Aardwolf MUSHclient Package Update Check Error:
Failed to load plugin network component.

Edit2: it looks like your aardwolf package is using a different lua 5.1.dll. Trying to substitute the standard lua.dll with the one from the package causes a lot of my lua code that is valid to break.

It's possible that llthreads requires luajit. I don't remember, though I see no reason why it would. You would have to check the github repository for that project to find out.

The luajit dll that I distribute is fully compatible with the Lua 5.1 dll distributed with MUSHclient. If functions aren't working then you either only copied the stub dll and not the real one (I believe you must take both either for luasec or llthreads because one of them uses the alternate canonical Lua dll name that MUSHclient.exe does not use) or you didn't take the vc100 redistributable dll which is also required due to compiler differences.

You can use a/the dependency walker to find exactly which files (DLLs) are missing.


http://www.dependencywalker.com/

Something like that came with Visual C++ and I used to use that in this sort of situation. It shows if there is a missing DLL, and sometimes it can be something not obvious, like one of the Microsoft library DLLs.

From my copy of Aardwolf client, here are the dependencies of Lua5.1.dll:

Hm, it seems that this version of the lua dlls have problems with the \ character. For instance,
f = io.input (GetInfo(56).."\worlds/ctips.txt")

needs to be changed to
f = io.input (GetInfo(56).."/worlds/ctips.txt")

another one:
local ret = os.execute("cd ..\.. & update_lite.exe")
needs to be
local ret = os.execute("cd ../.. & update_lite.exe")

Is it possible to make it work properly with the without having to replace the \ characters?

Edit: I've managed to get the dll to load, but it produces a error creating context when I try to get the results of the request. It works properly when I use the aardwolf client package though, so maybe I may have missed a step in the installation.

retval, page, status, headers, full_status = thread:join()
thread = nil
if status == 200 then -- 200 is the HTTP status code for "OK"
print("status "..status)
else
print("status is "..status)
end

I copied lua 5.1.dll, lua 51.dll, ssl.dll, async.lua, https.lua , ssl.lua and llthreads.dll from the aardwolf package. I didn't copy msvcr100.dll as it seems to affect screen reader's ability to work with mushclient; removing that from the aardwolf package makes it accessible, without breaking the ability to use the code above. I can't think of any steps that I missed.

In Lua, if you want a backslash inside a string you need to double it:



But do you really have a "worlds" folder in your root directory?

Just did some research on lua jit, and it looks like lua jit's just stricter on valid escape sequences.

Edit: I managed to figure out what was wrong, and managed to get everything working.

One (hopefully) last thing, is there a way for you to tell whether the request has been completed? I noticed that in your aardwolf package checker, the main() function is called using do after. I'm hoping this is possible, so that I can use this to implement support for a plugin updater.

Thanks fiendish for sharing this; I can now do https downloads, and it doesn't hang mushclient anymore.

Ah, now that you mention it I do remember also having that conversion speedbump when I switched over. Yeah, LuaJIT uses the Lua 5.2 strictness for invalid escape sequences.


It may be possible, but maybe not with the simple wrapper I made. I think you probably have to use interthread messaging via luasocket or zeromq for that. It's a good idea. I'll look into it later, but if you get it working before I get a chance I'd love to hear.


Cool. :)

Unfortunately, I'm no where nearly good enough to code something that complex. Look forward to seeing what you come up with :)

Hm, I wonder if this will work? It looks like a simpler solution to what you suggested.
https://github.com/moteus/lua-llthreads2

I think it might.
A new dll is compiled at
https://github.com/fiendish/aardwolfclientpackage/raw/development/MUSHclient/llthreads.dll

It should now let you check if thread:alive()
Let me know if it works for you?

[EDIT] amended original post with new dll location, because I haven't pushed it out of the development branch yet.

Wow, thanks a lot. It works :)

Hopefully you're still reading the thread. Can you clear this up for me? My understanding was that msvcr100.dll is necessary for my dlls to work because they are compiled with VC2010. How do they still load for you without it?

Maybe it's in his Windows/System32 directory.

Yup, its already in my windows/system32 directory. I think my version of the dll is different from the one you include Fiendish, because when I copied the one from my windows/system32 directory into the mushclient folder, it worked fine with screen readers.

Interesting. Can you send it to me somehow?

See: http://www.gammon.com.au/forum/?id=12520&reply=5#reply5

One of you could report to the other one the version, timestamp, size, etc. from the Dependency Walker.

Yeah but I want the file. :)

Thank you for this thread. I was having a similar issue and the answers here have been very helpful.

About llthreads, it can be linked statically, so it doesn't require the MSVCR100.dll file. I managed to do so by following the instructions here:
http://www.rhyous.com/2010/09/16/avoiding-the-msvcr100-dll-or-msvcr100d-dll
(The resulting .DLL is bigger, around 180k, but yet smaller than MSVCR100.dll.)

I recompiled/linked llthreads and luasec to work with lua5.1.dll, and they seem to be doing fine on the tests I've run so far.

File description Microsoft® C Runtime Library
Type Application extension
File version 10.0.40219.325
Product name Microsoft® Visual Studio® 2010

You can get this at https://dl.dropboxusercontent.com/u/2138237/msvcr100.dll

Since we can now check whether a thread has completed its job, the next step for me was to try to create something like a wait.lua that would allow you to pause execution and resume it when a thread completed. My attempts were pretty far from successful though :(

I got this working with the vanilla version of llthreads, which doesn't support thread:alive(). I make a list of files that require to be updated, based on a Dropbox file that has a list of current versions, and queue/download them ok.

I can share the code if you want, though it's not necessarily pretty. It uses global variables and a timer.

I'm pretty sure there's a more elegant solution, which of course is eluding me. :)

Are you polling file timestamps/hashes, or do you only work with files that contain their own version information?

At this moment I'm rewriting the scripts, so we're in transition.

The old version is very primitive. It only supports plugins that include 2 functions, called via CallPlugin(), to check which files the plugin uses. So users needs to run an alias to update a particular plugin, or all of them, and the script doesn't check if the files in my Dropbox are actually newer.

For the new version, I'm keeping the modification timestamps in a text file in my Dropbox (auto-generated by a script every X minutes). The new script detects which plugins are outdated by checing the date_modified field in the XML header of plugins vs those timestamps. I still haven't decided how to handle other files (modules, images, etc.). For now, they're all updated, brute-force, but I'll probably use hashes when I get to that stage.


Also, I rebuilt llthreads.dll, and it's 72 kbytes, statically linked (so it doesn't require msvcr100.dll.) I had toggled some stuff for testing, which made the file bigger.