Slipping podcasts through a filter - podcast

My workplace filters our internet traffic by forcing us to go through a proxy, and unfortunately sites such as IT Conversations and Libsyn are blocked. However, mp3 files in general are not filtered, if they come from sites not on the proxy's blacklist.
So is there a website somewhere that will let me give it a URL and then download the MP3 at that URL and send it my way, thus slipping through the proxy?
Alternatively, is there some other easy way for me to get the mp3 files for these podcasts from work?
EDIT and UPDATE: Since I've gotten downvoted a few times, perhaps I should explain/justify my situation. I'm a contractor working at a government facility, and we use some commercial filtering software which is very aggressive and overzealous. My boss is fine with me listening to podcasts at work and is fine with me circumventing the proxy filtering, and doesn't want to deal with the significant red tape (it's the government after all) associated with getting the IT department to make an exception for IT Conversations or the Java Posse, etc. So I feel that this is an important and relevant question for programmers.
Unfortunately, all of the proxy websites for bypassing web filters have also been blocked, so I may have to download the podcasts I like at home in advance and then bring them into work. If can tell me about a lesser-known service I can try which might not be blocked, I'd appreciate it.

Can you SSH out? SSH Tunnels are your friend!

Why not subscribe at home and have your favorite podcasts copied to your mp3 player or a USB drive and just take it to work with you each day and back home in the evening? Then you can listen and your are not circumventing your clients network.

There are many other Development/Dotnet/Technology podcasts, try one of those. for the blocked sites try an anonymous proxy site, there are plenty out there.

Since this is work related material, I would recommend opening up a request that the sites in question not be blocked.

I ended up writing an extremely dumb-and-simple cgi-script and hosting it on my web server, with a script on my work computer to get at it. Here's the CGI script:
#!/usr/local/bin/python
import cgitb; cgitb.enable()
import cgi
from urllib2 import urlopen
def tohex(data):
return "".join(hex(ord(char))[2:].rjust(2,"0") for char in data)
def fromhex(encoded):
data = ""
while encoded:
data += chr(int(encoded[:2], 16))
encoded = encoded[2:]
return data
if __name__=="__main__":
print("Content-type: text/plain")
print("")
url = fromhex( cgi.FieldStorage()["target"].value )
contents = urlopen(url).read()
for i in range(len(contents)/40+1):
print( tohex(contents[40*i:40*i+40]) )
and here's the client script used to download the podcasts:
#!/usr/bin/env python2.6
import os
from sys import argv
from urllib2 import build_opener, ProxyHandler
if os.fork():
exit()
def tohex(data):
return "".join(hex(ord(char))[2:].rjust(2,"0") for char in data)
def fromhex(encoded):
data = ""
while encoded:
data += chr(int(encoded[:2], 16))
encoded = encoded[2:]
return data
if __name__=="__main__":
if len(argv) < 2:
print("usage: %s URL [FILENAME]" % argv[0])
quit()
os.chdir("/home/courtwright/mp3s")
url = "http://example.com/cgi-bin/hex.py?target=%s" % tohex(argv[1])
fname = argv[2] if len(argv)>2 else argv[1].split("/")[-1]
with open(fname, "wb") as dest:
for line in build_opener(ProxyHandler({"http":"proxy.example.com:8080"})).open(url):
dest.write( fromhex(line.strip()) )
dest.flush()

Related

watir-webdriver cookie jar saving and loading

I'm surprised there didn't seem to be much discussion on this.
in Mechanize I can easily read an entire cookie jar from the browser, store it to a file, and load it in to a later session/run before loading that website's pages again.
How can one do the same with watir-webdriver?
UPDATE
Now with 0.5.2 I do see new methods browser.cookies.to_hash
which would turn this question into "How to implement .from_hash or similar loader using eg. .clear and .add?"
However I'd be especially keen on loading and saving all cookies using previous versions (0.4.1) which my servers are likely to be stuck with for a while. Via the Selenium driver maybe?
browser = Watir::Browser.new :firefox
browser.goto 'http://google.com'
# save cookies
saved_cookies = browser.cookies.to_a
# clear and get new cookies
browser.cookies.clear
browser.goto 'http://google.com'
# set new cookies
browser.cookies.clear
saved_cookies.each do |saved_cookie|
browser.cookies.add(saved_cookie[:name], saved_cookie[:value])
end
Applying pOdeje's loop to repopulate the cookie jar, here's a solution that includes storing the cookies into a file, readable in a later Ruby run. A straight File.open write and read of the array had some issues I didn't wish to work around (parsing?), but YAML object serialization already bundled in Ruby was well suited to the task.
require 'yaml'
# Save/serialize cookies
# File.open("ST.cookies.txt", 'w').write $browser.cookies.to_a.to_s
File.open("ST.cookies.yaml", 'w').write YAML::dump($browser.cookies.to_a)
# Load/deserialize cookies
# $cookies = File.open("ST.cookies.txt", 'r').to_a # returns 1-elem array of single long line, not indiv elements
$cookies = YAML::load(File.open("ST.cookies.yaml", 'r'))
$browser.cookies.clear
$cookies.each do |saved_cookie|
$browser.cookies.add saved_cookie[:name],
saved_cookie[:value],
:domain => saved_cookie[:domain],
:expires => saved_cookie[:expires],
:path => saved_cookie[:path],
:secure => saved_cookie[:secure]
end
Still on the lookout for a pre watir-webdriver 0.5.x method, though.
CAVEAT
Not thoroughly tested yet but it seems I have to first load the URL to which the cookies apply, then load in my cookie jar using the above method, and finally load that URL into $browser object a second time. This is only a minor inconvenience and time cost for my case in which I'm staying within the same domain throughout my web session, but I can see this turning into a real thorn for cookie jars affecting several unrelated sites (as had indeed been an expectation for my old programs using other languages & libraries like Mechanize). Curl and Wget and generally other tools I've used for SOAP interfacing always let me control my POST/session/cookie environments prior to site loading. Just a thought.

Problems with MSF4J and #MatrixParam

Folks, I have found what seems to be a problem with / (bug in ?) MSF4J as including an #MatrixParam annotated variable in a URI causes the affected (micro)service to either 'hang' indefinitely, or if accessed via a browser, to give a "404 Not Found" message for the path/endpoint, even when correct.
Here is a code fragment that illustrates the problem - it compiles ok (eclipse/maven) and deploys without errors using microservicesrunner() in the usual way.
package org.test.service;
import javax.ws.rs.GET;
import javax.ws.rs.MatrixParam;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
#Path("books")
public class MPTest { // MatrixParam Test
#GET
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
// method to respond to 'GET' requests
public Response getListOfBooks(#MatrixParam("Author") String author) {
// do something in here to get book data from DB and sort by titles
List<String> titles = .......;
return Response.status(200) .entity("List of Books by " +author+ "ordered by title " + titles).build();
}
}
With this code fragment, accessing the URL "(host:8080)/books/query;Author=MickeyMouse" should cause a list of books by that author to be retrieved from the DB (I have omitted the actual code that does so for clarity, as it is not relevant to this post).
However, it does not get there, so that code isnt executed. As far as I can tell with a debugger, no #MatricParam value is retrieved - it remains null until the process times out. Things like curl and wget just hang until they time out, and from a browser, the best I can get is a 404 not found error for the URI, even though it is valid.
However, if I replace the #MatrixParam with a #PathParam it works perfectly, and can I get the URL string retrieved in its entirity. The URI that I get is as expected - no odd hex characters, no typos, and so forth. The URI entered is what you get back. So, no problem there.
Behaviour is also consistent across platforms (couple of flavours of Linux, and three versions of Windoze), so it is not anything to do with the OS itself. Similarly, I get the same behavior with multiple clients and tools, so it isnt a problem there either.
So, it appears to be a problem within the MSF4J framework / domain, and I could use some support / help / suggestions here as I've reached the point of tearing my hair out..... Any ideas, folks?
The only reference I can find to a similar problem was closed as 'off topic' without a reply (see Rest API Matrix param annotation) so I think that this needs re-opening as it seems to be a genuine problem....
Regards, and thanks in advance for any help,
Rick
#MatrixParam is not supported with MSF4J at the moment. You can create a GitHub issue. So we can implement that support in future releases.

Rewrite dynamic URLs in lots of HTML files

I am working on a migration project form a legacy Intranet to a new product.
All the html files are stored on a file system but will be chucked into a database and I am cleansing a lot of HTML already, what I need to do now is to rewrite URLs so that they will continue to work
At the moment we have thousands of files with query string links like so:
Hanoi
Ho Chi Minh City
Hong Kong
I need something to alter all the links to instead look like:
Hanoi
Ho Chi Minh City
Hong Kong
The number will always be a three digit number
I'm already doing a lot of other transformation in the HTML but got stuck on this one
Any ideas?
Thanks in advance
Edit: the pages will be migrated to SharePoint and hence be stored in a SharePoint content database for internal use.
Edit2:
This solved the problem but can it be done neater?
public static string ReplacePageUrl(string content)
{
string updatedContent = content;
for (int i = 1; i < 1000; i++)
{
updatedContent = updatedContent.Replace("site.get?section&PAGE" + i.ToString("D3"),
"href=\"/sites/pages/page" + i.ToString("D3") + ".aspx");
}
return updatedContent;
}
modifying the database and keep a single instance of it is quite dangerous and it could be error prone, in the case you are doing URL modifications try the following scenario:
Do a 301 redirect:
The HTTP response status code 301 Moved Permanently is used for
permanent redirection, meaning current links or records using the URL
that the 301 Moved Permanently response is received for should be
updated to the new URL provided in the Location field of the response. - Wikipedia
by doing this you aren't going to lose whatever the ranking you have on your current pages and it won't affect SEO at all.
Create or copy the new content to another Database
Once you have the list of urls you are going to redirect the you can perform the url changes on this database (the new url structure) and then you can analyze and study the redirections and do all the necesary modifications to this database without having broken links.
Test the redirections
I would suggest to test your redirections prior to launch your product to production (this sounds quite obvious isn't it) but make sure that you are modifying your local ETC file system to point the domain of your application to the 127.0.0.1 ip
Run several tests on google, write whatever the url pattern you are testing to a google query like "page1.aspx?param=1" site:yourdomain.com and try every single pattern / redirection just to make sure eveything is working.
I think that a safer way to work and it could give you the flexibility to at least miss a couple of links without having broken links.
EDIT:
Well if you need to replace or modify HTML and that's your only concern I'd recommend you to use HTML Agility Pack you can do linq queries to the HTMl elements doing filters, elements modifications, that will make your scenario a lot easier!
something like this:
var links = html.DocumentNode
.Descendants("tr")
.Where(tr => tr.GetAttributeValue("class", "").Contains("alt"))
.SelectMany(tr => tr.Descendants("a"))
.ToArray();
I'd recommend Notepad++ (http://notepad-plus-plus.org) to replace the text in your files.
Then you can use the "Replace in files" function with a regular expression...
With regular expressions enabled in the find/replace box, replace this:
(.*)
With this:
\2

JFrame in remote between JDK 5 (Server) and 6 (Client - VisualVM)

So I have a little trouble on the opening of a JFrame. I searched extensively on the net, but I really can not find a solution ...
I explained the situation:
I need to develop an application that needs to retrieve information tracking application while meeting new safety standards. For that I use JMX that allows monitoring and VisualVM to see these information.
I therefore I connect without problems (recently ^ ^) to JMX since VisualVM.
There is thus in a VisualVM plugin for recovering information on MBean, including those on Methods (Operations tab in the plugin).
This allows among others to stop a service or create an event.
My problem then comes when I try to display a result of statistics.
In fact, I must show, at the click of a button from the list of methods in the "Operations", a window with a table in HTML (titles, colors and everything else).
For that I use a JFrame:
public JFrame displayHTMLJFrame(String HTML, String title){
JFrame fen = new JFrame();
fen.setSize(1000, 800);
fen.setTitle(title);
JEditorPane pan = new JEditorPane();
pan.setEditorKit(new HTMLEditorKit());
pan.setEditable(false);
pan.setText(HTML);
fen.add(pan);
return fen;
}
I call it in my method:
public JFrame displayHtmlSqlStatOK_VM(){
return displayHTMLJFrame(displaySQLStat(sqlStatOK, firstMessageDate), "SqlStatOK");
}
The method must therefore giving me back my JFrame, but she generates an error:
Problem invoking displayHtmlSqlStatOK_VM : java.rmi.UnmarshalException: error unmarshalling return; nested
exception is:
java.io.InvalidClassException: javax.swing.JFrame; local class incompatible: stream classdesc serialVersionUID =
-5208364155946320552, local class serialVersionUID = -2386951414768123374
I saw on the internet that this was a version problem (Serialization), and I believe strongly that it comes from the fact that I have this:
Server - JDK5 <----> Client (VisualVM) - JDK6
Knowing that I can not to change the server version (costs too important ...) as advocated by some sites and forums.
My question is as follows:
Can I display this damn window keeping my current architecture (JDK5 server side and client side JDK6)?
I could maybe force the issue? Tell him that there's nothing bad that can run my code? Finally I'm asking him but he does not answer me maybe to you he will tell you ... (Yes I crack ^^).
Thank you very much to those who read me and help me!
If you need more info do not hesitate.
EDIT
The solution to my problem might be elsewhere, because in fact I just want a table with minimal formatting (this is just for viewing application for an for an officer to have his little table him possibly putting critical data in red...).
But I have nowhere found a list of types that I can return with VisualVM ... This does not however seem to me too much to ask.
After I had thought of a backup solution, which would be to create a temporary HTML file and open it automatically in the browser, but right after that is perhaps not very clean ... But if it can work ^^
I am open to any area of ​​research!
It looks like you are sending instance javax.swing.JFrame over the JMX connection - this is a bad idea.
Well good I found myself, as a great :)
Thank you bye!
..........
Just kidding of course I will give the solution that I found ^ ^
So here's what I did:
My display to be done on the client (normal...) my code to display a JFrame that I had set up on the server was displayed obviously ... On the server xD
I didn't want to change the customer (VisualVM) to allow users maximum flexibility. However I realized that to display my HTML table to be rendered usable (with colors and everything) I had to change the client (as JMX does not support the type JFrame as type back an operation).
My operation running from the MBeans plugin for VisualVM, it was necessary that I find the source code for it to say "Be careful if you see that I give you the HTML you display it in a JFrame".
Here is my approach:
- Get the sources
The link SVN to get sources VisualVM is as follows:
https: //svn.java.net/svn/visualvm~svn/branches/release134
If like me you have trouble with the SVN client includes in NetBeans because you are behind a proxy, you can do it by command line:
svn --config-option servers:global:http-proxy-host=MY_PROXY_HOST --config-option servers:global:http-proxy-port=MY_PROXY_PORT checkout https: //svn.java.net/svn/visualvm~svn/branches/release134 sources-visualvm
Putting you on your destination folder of course (cd C:\Users\me\Documents\SourcesVisualVM example).
- Adding the platform VisualVM
NetBeans needs the platform VisualVM to create modules (plugins) for it. For this, go to "Tools" -> "NetBeans Platforms".
Then click "Add Platform ..." at the bottom left of the window and select the folder to the bin downloaded at this address: http:// visualvm.java.net/download.html
You should have this:
http://img15.hostingpics.net/pics/543268screen1.png
- Adding sources in the workspace (NetBeansProjects)
Copy/paste downloaded sources (SVN from the link above) to your NetBeans workspace (by default in C:\Users\XXX\Documents\NetBeansProjects).
- Ouverture du projet du plugin MBeans
In NetBeans, right click in the Project Explorer (or go to the menu "Files") and click "Open Project ...".
You will then have a list of projects in your workspace.
Open the project "mbeans" found in "release134" -> "Plugins", as below:
http://img15.hostingpics.net/pics/310487screen2.png
- Changing the file "platform.properties"
To build plugin you must define some variables for your platform.
To do this, open the file platform.properties in the directory release134\plugins\nbproject of your workspace.
Replace the content (by changing the paths compared to yours):
cluster.path=\
C:\\Program Files\\java6\\visualvm_134\\platform:\
C:\\Program Files\\java6\\visualvm_134\\profiler
# Deprecated since 5.0u1; for compatibility with 5.0:
disabled.clusters=
nbjdk.active=default
nbplatform.active=VisualVM_1.3.4
suite.dir=${basedir}
harness.dir= C:\\Program Files\\NetBeans 7.1.2\\harness
- Changing the class XMBeanOperations
To add our feature (displaying an HTML table), you must change the class that processes operations, namely the class XMBeanOperations in package com.sun.tools.visualvm . modules.mbeans.
At line 173, replace:
if (entryIf.getReturnType() != null &&
!entryIf.getReturnType().equals(Void.TYPE.getName()) &&
!entryIf.getReturnType().equals(Void.class.getName()))
fireChangedNotification(OPERATION_INVOCATION_EVENT, button, result);
By :
if (entryIf.getReturnType() != null &&
!entryIf.getReturnType().equals(Void.TYPE.getName()) &&
!entryIf.getReturnType().equals(Void.class.getName())) {
if (entryIf.getReturnType() instanceof String) {
String res = result + "";
if (res.indexOf("<html>") != -1) {
JFrame frame = displayHTMLJFrame(res, button.getText());
frame.setVisible(true);
}
else
fireChangedNotification(OPERATION_INVOCATION_EVENT, button, result);
} else
fireChangedNotification(OPERATION_INVOCATION_EVENT, button, result);
}
With the method of creating the JFrame that you place above "void performInvokeRequest (final JButton button)" for example:
// Display a frame with HTML code
public JFrame displayHTMLJFrame(String HTML, String title){
JFrame fen = new JFrame();
fen.setSize(1000, 800);
fen.setTitle(title);
JEditorPane pan = new JEditorPane();
pan.setEditorKit(new HTMLEditorKit());
pan.setEditable(false);
pan.setText(HTML);
fen.add(pan);
return fen;
}
We can see that we already did a test on the return type, if it is a String which is returned, if the case, if we see in this string the balise , then we replace the result of the click by opening a JFrame with the string you put in, what makes us display our HTML code!
- Creating a .nbm
The file .nbm is the deployment file of your plugin. Simply right-click your project (in the Project Explorer) and click on "Create NBM".
Your file .nbm will be created in the folder "build" the root of your project.
- Installing the plugin in VisualVM
To install your plugin, you must just go in VisualVM, go into "Tools" -> "Plugins" tab and then "Downloaded", click "Add Plugins ...". Select your plugin .nbm then click "Install". Then follow the instructions.
Useful Sources
http: //docs.oracle.com/javase/6/docs/technotes/guides/visualvm/
http: //visualvm.java.net/"]http://visualvm.java.net/
http: //visualvm.java.net/api-quickstart.html (Créer un plugin VisualVM avec NetBeans)
Thank you very much for your help Tomas Hurka ;)

C# upload file to server - both client and server side

I'm a C# game programmer with little web development experience.
I need to upload a small file (25-100 or so bytes, depending on it's content) to a server. This is on the Windows Phone 7 using XNA. The target server is fairly limited and only supports PHP and classic ASP.
Since the CF on the WP7 only has access to a limited subset of networking commands, it's looking like an HttpWebRequest GET aimed at a script that saves the file will be the best option. The data I'm sending is small in size, and should be able to be passed as a parameter in the url.
I've been searching but have yet to find a complete example of this, which handles both the client and server side script (mainly the latter). This is close to what I'm looking for, except it has no mention of the server side script: Upload files with HTTPWebrequest (multipart/form-data)
The closest that I got was this: http://www.johny.org/2007/08/upload-using-c-as-client-and-php-as-server/
But when attempting to use it I get an unhandled exception: "The remote server returned an error: (405) Method Not Allowed". This method seems the most promising so far, but I've yet to be able to debug this.
Unfortunately, I have a short amount of time to implement this, and as I said only a passing familiarity with web development. I'm not worried about maximum security or scalability as this is a temporary measure to collect feedback internally. Basically, I just need the quickest thing that works. ;)
Any help would be fantastic!
I've solved it. First off, PHP wasn't supported on my server (just now learning that PHP and ASP are can't be used on the same server, depending on whether it's on Linux or Windows - like I said, web development noob here!). I switched to ASP and, after digging through the docs, wrote this script:
<%
dim theData, theFileName
set theData=Request("data")
set theFileName=Request("filename")
dim fs,tfile
set fs=Server.CreateObject("Scripting.FileSystemObject")
set tfile=fs.CreateTextFile(Server.MapPath(theFileName+".txt"))
tfile.WriteLine(theData)
tfile.Close
set fname=nothing
set fs=nothing
set theData=nothing
set theFileName=nothing
%>
This C# code uploads the file:
const string cAddress = "http://site.com/folder/upload.asp";
string fileName = foo;
string data = bar;
string address = cAddress + "?filename=" + fileName + "&data=" + data;
uploadRequest = (HttpWebRequest) HttpWebRequest.Create(address);
uploadRequest.Method = "GET";
uploadRequest.GetResponse();
Hope this helps someone else looking for an example of how to do this!
But you have the METHOD as GET instead of POST. You can't upload a file to a website by passing the file path to the Query String.

Resources