Assuming it's not malfunctioning then when you're not using the Slingbox (ie your not watching the stream from somewhere) then it won't have any impact on the Internet connection where it is located.
If you are streaming, then it may utilise up to about 8mbps of the upload speed. If you saturate the upload in this way it may well have a negative effect on the download speed for people using the Internet at the Slingbox location. What is the upload speed at the location of the Slingbox and do you typically use all of this when streaming?
If they are complaining at the Internet speed during times you are not streaming then barring some obscure malfunction on the part of the box it won't be the cause of it.
The easiest way to tell of course is if they temporarily unplug the network cable from the Slingbox at these times and see if it makes any difference!