Discussion about JeeLabs projects
EtherCard trouble
I'm guessing it has something to do with this packetLoop magic, but I just can't figure it out. The first call to _httpRequest succeeds, but when I call it again immediately it times out,
Problem number two is that after x failed requests I try to restart the network by first resetting the ENC card, calling ether.begin and then ether.dhcpSetup. But the call to dhcpSetup always fail in this case.
Thanks in advance for any input on these two problems, I'm kind of stuck..
Anders
#define HTTP_READ_TO 5000
extern prog_char backendServer[];
static boolean httpHaveReply = false;
static word httpReplyOffset;
static byte httpReplyStatus;
static void _onHttpReply(byte status, word off, word len) {
httpReplyOffset = off;
httpReplyStatus = status;
httpHaveReply = true;
}
// Make HTTP GET request and return last line of output
// NOTE: buf is used for both input and output
int _httpRequest(char *buf, int bufLen) {
httpHaveReply = false;
httpReplyStatus = 1;
int ret = BE_ERROR;
// Send request
dbg("-> %s" NEWL, buf);
while (ether.packetLoop(ether.packetReceive()) != 0) {}
ether.browseUrl(PSTR("/api/"), buf, backendServer, _onHttpReply);
dbg("-> sent");
// Wait for reply
MilliTimer t;
t.set(HTTP_READ_TO);
while (!httpHaveReply) {
ether.packetLoop(ether.packetReceive());
if (t.poll()) {
dbg(" timeout");
ret = BE_NETWORK_ERROR;
break;
}
}
buf[0] = '\0';
// Check result and extract last line of output
if (httpReplyStatus == 0) {
dbg(" 200");
char *p = (char *)Ethernet::buffer + httpReplyOffset;
while (*p) {
while (*p) if (*(p++) == '\n') break;
if (*p == 0) break; // run out of buffer
if (*(p++) == '\r') break;
}
if (*p) {
p++; // skip \n
strncpy(buf, p, bufLen);
}
dbg(" [%s] done" NEWL, buf);
ret = BE_SUCCESS;
} else {
dbg(" fail" NEWL);
}
return ret;
}
It's odd how the EtherCard library works for some and not for others. Probably you already did - but make sure you always check for updates - jeelabs.net/projects/cafe/repository/show/EtherCard - the ZIP archive is also automatically rebuilt, in case you're not using subversion.
I'm still poking here and there to try and improve things. All I can say is that the example sketches all work here. With DHCP there may well be some issues with specific brands of routers (i.e. their DHCP servers, to be exact).
I can review Wireshark dumps, and compare to mine - don't know how else to help out here. When all else fails, my suggestion would be to simplify until we can figure it out. Are any of the example sketches working for you?
Thanks for the reply JC. The examples all work fine, although I've seen some unexplainable hangs and timeouts in HTTP requests and DHCP. DHCP works in my code when used in setup(), but I can't use it again for some reason even after resetting the ENC chip. I will need to do some wiresharking to figure this one out.
I think I have found the problem with calling browseUrl repeatedly. Since it uses a global variable for www_fd, stray TCP replies for a previous request may trigger the callback armed for the current request (status will be 4)
static byte www_client_internal_result_cb(byte fd, byte statuscode, word datapos, word len_of_data) {
if (fd!=www_fd)
(*client_browser_cb)(4,0,0);
At least I think that is what is going on.. I'm rewriting my code to use clientTcpReq directly which returns the fd, which can be used to identify replies in the callback function. I'll post any progress here.
The below code seems to work better, for me at least. Basically the reply callback ignores any packets not related to the most recent HTTP request. Maybe I will convert it to a small class to get rid of the global variables, if I can figure how to pass C++ private methods as callbacks :)
Just to give you some background on what I'm trying to achieve. I have a circular buffer of measurements waiting to be uploaded to a remote server using HTTP GET (yes you REST fanatics, I will change this to POST later :). After a network outage _httpRequest must be called repeatedly without returning to loop, since the loop also processes inbound HTTP requests and some other stuff which I think would garble the communication. This is why I needed a blocking request function and apparently also this matching of requests and replies.
#define HTTP_READ_TO 5000
extern prog_char backendServer[];
extern const int backendPort;
extern BufferFiller bfill;
static boolean httpHaveReply = false;
static word httpReplyOffset;
static byte httpReplyStatus;
static byte httpRequestFd;
static char *httpRequestUrl;
static word _httpDatafillCb(byte fd) {
bfill = ether.tcpOffset();
bfill.emit_p(PSTR("GET /api/$S HTTP/1.1\r\n"
"Host: $F\r\n"
"Accept: text/html\r\n"
"Connection: close\r\n"
"\r\n"), httpRequestUrl,
backendServer);
return bfill.position();
}
static byte _onHttpReply(byte fd, byte status, word off, word len) {
if (fd == httpRequestFd) {
httpReplyOffset = off;
httpReplyStatus = status;
httpHaveReply = true;
}
}
// Make HTTP GET request and return last line of output
// NOTE: buf is used for both input and output
int _httpRequest(char *buf, int bufLen) {
httpHaveReply = false;
httpReplyStatus = 1;
httpRequestUrl = buf;
int ret = BE_ERROR;
// Send request
dbg("-> %s" NEWL, buf);
while (ether.packetLoop(ether.packetReceive()) != 0) {}
dbg("->");
httpRequestFd= ether.clientTcpReq(_onHttpReply, _httpDatafillCb, backendPort);
dbg(" sent");
// Wait for reply
MilliTimer t;
t.set(HTTP_READ_TO);
while (!httpHaveReply) {
ether.packetLoop(ether.packetReceive());
if (t.poll()) {
dbg(" timeout" NEWL);
return BE_NETWORK_ERROR;
}
}
buf[0] = '\0';
// Check result and extract last line of output
if (httpReplyStatus == 0) {
dbg(" 200");
char *p = (char *)Ethernet::buffer + httpReplyOffset;
while (*p) {
while (*p) if (*(p++) == '\n') break;
if (*p == 0) break; // run out of buffer
if (*(p++) == '\r') break;
}
if (*p) {
p++; // skip \n
strncpy(buf, p, bufLen);
}
dbg(" [%s] done" NEWL, buf);
ret = BE_SUCCESS;
} else {
dbg(" fail (%i)" NEWL, httpReplyStatus);
}
return ret;
}
There's a lot of nastiness in the underlying code, which is still based on the original Stang/Socher design. But I'm not going to point the finger at them, because I made some huge sweeping changes across the board in an attempt to "simplify" things. I may well have messed up various cases.
Only way to climb out of this hole, is to try and fix problems as they pop up. So please do keep reporting them.
After a network outage _httpRequest must be called repeatedly without returning to loop, since the loop also processes inbound HTTP requests and some other stuff which I think would garble the communication.
You could set another (hmm...) global flag, which tells the main loop not to do any incoming work, until the outgoing request has been fully handled. Not great, but perhaps good enough for this scenario?
Isn't multi-tasking "fun" when there is no multi-tasker?
Another option would be to look at the Scheduler class in the Ports library - jeelabs.org/2010/10/17/scheduling-multiple-tasks/
Thanks, I'll take a look at the Scheduler. However repeated HTTP-requests seem to work fine now, with the code posted above. I also figured out the problem with calling dhcpSetup twice - I just had to zero out ether.myip before calling it. Doh!
I think your strive to hide some of the complexity of the underlying TCP/IP-code is very useful for a lot of people, so keep on it. You're doing a great job!
BTW, I have another EtherCard related question. I understand that it is crucial to keep the entire payload inside a single TCP-packet when using EtherCard (well, any ENC-based network device really). I see that the MSS is #defined in tcpip.cpp to 550, which is what is sent in the initial SYN-packet. But the decision to put the data (for example a HTTP reply) in separate TCP packets or not is up to the server's IP-stack - isn't it? The reason for asking is that I've been doing a lot of Wiresharking today :) and I see that my backend (Google App Engine) sometimes puts the response in multiple TCP packets, even if the total length is nowhere near 550 bytes. Is there a way to avoid this?
I've added the myip clearing to dhcpSetup() - thanks.
Yes, I think it's up to the sender to decide how to fragment its transmission, as long as it stays under the MTU limit agreed upon during the initial handshake.
The single-packet limit is just a software limitation really - I don't know whether it applies to both sides of the connection, and what exactly would have to be changed to remove this limitation. That, and the packet-buffer-in-RAM requirement are the main limitations of this whole setup (both could be overcome with better code on the ATmega, the ENC chip is not the limiting factor). Fortunately, it's still workable for low-end use.
Andersw, jcw
We have experienced router specific problems with the same ethershield library - we use for Nanode.
My Netgear DGN1000 refuses to behave correctly to DHCP requests, and my Linksys also had problems until Andrew Lindsay upgraded the library.
We are currently compiling a list of known router problems and hope to find solutions
Ken
I've had some problems with the DHCP functions also. What I did notice is that the first DHCP request is being sent before the link is even up :) We should check for link up before starting a DHCP request. After doing that I at least saw the request in my packet sniffer :)
Ah, good catch! I've made a small change to dhcp.cpp and dns.cpp - does that fix it?
It does not, at least for me. @monsonite sent me a Nanode, so I've been running with that. All is well except DHCP is flaky. I have 7752. Also I've been using EtherCard for a few months now successfully, but have only just now started to try DHCP. Sometimes it works, sometimes not.
What's the status of this right now? DHCP with EtherCard lib is flakey - how about DHCP with Andy's EtherShield library? It'd be nice to get things resolved.
The EtherCard DHCP client seems very stable here, using Zyxel P-660HW-D1 ADSL router.
I also found the cause of the problem I experienced with the HTTP client earlier... A bit embarrassing actually - turned out that I didn't provide enough power to the EtherCard. I used a USB hub with a PSU that should give out 500mA, but actually didn't. After connecting the Jeenode to an external PSU things started to work much better.
A spectacular amount of hours wasted on that one I can tell you :)