Last week, we told you about peer-to-peer and torrent file-sharing sites were being systematically shut down all over China . Not too long before that, we let you know about file-sharing being monitored by a major ISP in the UK. Now, Israeli ISPs are throttling P2P network access, too, as confirmed in a report just released by an Israeli cyberlaw attorney and a partner news site. Whether you consider file-sharing an affront to content creators and copyright-holders everywhere or whether you see P2P networks as a permissible and valid way for users to exchange data, this trend is gaining considerable momentum around the world. Where will P2P restrictions pop up next? Sponsor In their research, tech attorney Jonathan Klinger and researchers involved with the Israeli website Ynet found that two of the three major ISPs in Israel are interfering with user traffic and might be conducting deep packet inspection. Traffic shaping is a practice sometimes used by some ISPs to discourage the use of certain applications. A couple of years ago, Comcast caught some heat from users and media for filtering user traffic when torrent files were being downloaded, even causing some to speculate that the ISP was violating U.S. law by prohibiting this traffic. Eventually, Comcast did strike a deal with BitTorrent to allow protocol-agnostic traffic management, but only after the sparring had been brought to the attention of the Federal Communications Commission. It’s currently unclear whether Israeli ISPs are filtering traffic due to piracy concerns or simple due to bandwidth concerns, as shared files can often amount to multi-gigabyte, hours-long downloads. However, traffic-shaping that blocks P2P protocols can also apply to VoIP calls, IM clients and other applications. Although P2P traffic is associated with illegal downloads, nothing about the protocols themselves is inherently illegal. “The element common to all P2P services,” reads the Israeli report, “is the lack of economical benefit to the ISP.” Klinger noted that although complaints have been brought to media outlets and ISPs since 2007, the ISPs have typically ignored these criticisms. Netvision and Internet Zahav were the two ISPs determined by this research to be blocking file-sharing traffic. Bezeq International was the third ISP investigated. Although Bezeq was cleared by this particular investigation, a plug-in introduced last year from popular bittorrent client Vuze shows that this ISP, too, throttles and disrupts file-sharing network traffic. In response to the findings presented by Ynet and Klinger, all three of the investigated ISPs gave typically canned responses claiming to offer users excellent surfing experiences. Israeli Communications Ministry rep Dr. Yechiel Shabi told Ynet, “The research materials relayed to us paint a picture which arouses the need for thorough examination. After we become familiar with the study’s findings, we shall consider the need for interference, supervision or regulation of the matter.” So, while we wait to see what results this report will yield in Israel, we are left to ponder the perturbing question: Where will traffic-shaping pop up next to prevent P2P activity? Take another look at the findings from Vuze’s traffic-monitoring plug-in . You’ll see that ISPs around the world – including Verizon, BellSouth, AOL, AT&T, Charter, Road Runner and ISPs in France, Italy, Germany, Poland, the UK and the Middle East, to name a few locations – are already interrupting traffic. Vuze’s researched was released in April 2008; in August, the FCC declared that ISPs should not be allowed to target and interrupt P2P applications. Still, suspicious Americans and other users around the world should consider using a tool such as the EFF’s Switzerland to determine whether torrent downloads and VoIP calls are being interrupted by their ISP. Do Israeli or other ISPs have the right or the moral imperative to throttle traffic in this manner? Do they have the need or right to examine the applications, files, and protocols being employed by users on their networks? Or do ISPs around the globe need to read the wiki on net neutrality and get their act together? Let us know your experiences and opinions in the comments. Discuss

Go here to read the rest:
P2P Sharing Being Blocked Around the World, Where Next?

2009 has been a turning point for the Internet of Things , when real world objects (such as lights, cars and packages) get connected to the Internet. This trend has added a significant amount of new data to the Web, so for that reason alone it is an important development. Having said that, many of the following top 10 list are not yet mainstream products. But we expect some of them to become well known over the coming years. Underlying the Internet of Things are technologies such as RFID (radio frequency identification), sensors

About 18 months ago, we wrote about an obscure search startup from Germany called FAROO . We believed that its radical alternative, using peer-to-peer (P2P) technology, had a shot at being a real disruptive force. Today, it has made some progress, has raised some money and is getting out into the market. (Disclosure: FAROO is currently a ReadWriteWeb sponsor). FAROO is wisely underplaying P2P in its marketing, preferring more fashionable terms such as “real-time search” and “social discovery.” But the P2P technology drives it. Sponsor So, we decided to invite someone who understands P2P at a technical level to interview Wolf Garbe, FAROO’s founder. Our tech expert, Kiril Pertsev, of Agily Networks , has already written about P2P for us in the past . Kiril: Why .NET? Did you already have development resources or did you make this choice because you consider it a better option for networked desktop applications? Would you make this choice again? And if you’re not satisfied with .NET, what would your platform of choice be, given all of your experience over the past few years? Wolf: I come from Delphi (Object Pascal). So, the choice of C#/.NET was a dedicated decision for a new platform, not driven by legacy. When I started to work on the first prototype in 2004, Delphi moved towards .NET. I preferred to go with the original, especially because the development of C# was led by Anders Hejlsberg, the designer of Borland’s Turbo Pascal (which Delphi derived from). Of course, I also looked into Java, which I found quite similar, both from the language perspective (C# vs. Java) and the JIT Runtime environment (Java Virtual Machine vs. .NET Runtime). The decision for C# was based on the dominating desktop market share of Windows and the assumption that embedding the .NET framework into the OS would ensure fast penetration of .NET. This only partially came true, partly due to the limited success of Vista, which was the first Windows version with .NET pre-installed. Kiril: Doesn’t this choice hinder your ability to move to Mac and Linux platforms. Wolf: We were betting on Mono for platform compatibility. Unfortunately, Mac OS X still has no Mono application launcher, other than starting with the terminal, which is not feasible for a mass market. With the increasing importance of the Mac OS X platform, I expect this to change. Silverlight today is already natively available for Mac. For the ultimate platform independence, we are also continually observing the diverse RIA developments (AJAX, AIR, Silverlight, Mozilla Prism, HTML 5 persistent web storage, Mozilla’s DOM storage, Google Gears and Flash persistent storage), which could one day allow us to remove the download and installation step for P2P. But so far, no solution meets all of the requirements: out-of-browser capability, permanent background operation, auto-start option, tray icon support, cross-domain connection support, persistent storage, accepting an incoming connection and receiving data and NAT traversal. Kiril: If you become dissatisfied with .NET, what would be your next platform of choice. Wolf: Although not everything went as expected, I still believe that .NET is a very powerful platform, and C# as a language is evolving at a much faster and broader pace than Java. Today, we have a good .NET penetration rate in the US and Europe. With Windows 7, I expect that to increase in Asia as well. Kiril: I see that you’re using a pretty simple P2P communication technology instead of sophisticated Hamachi-like NAT traversal using UDP hole punching. Wolf: I suppose you are referring to the transport layer, which is HTTP over TCP/IP. The real P2P overlay protocol on top of that is not that simple anymore. Because our distributed search engine system architecture breaks with almost all legacy paradigms, we thought it would be a good idea that it be at least based on proven and widely used standards wherever possible. There are several reasons for this: It reduces complexity and development time. It improves compatibility (there is probably no protocol more widely used than HTTP over Port 80). It’s unlikely that this connectivity will break anytime soon by changes in protocols, OS, drivers or hardware. Behaving like a standard browser from the protocol view makes the application less vulnerable to filtering, blocking or traffic shaping and ensures that it even works in most corporate environments. NAT traversal is the most critical issue for every P2P application. It’s really a shame that although the Internet is built on a distributed foundation, end-to-end connectivity between users in a decentralized way is completely broken . We are using several NAT traversal techniques: Manual Port Forwarding, Automatic Port Forwarding via UPnP and Teredo. Teredo is a IPv6 Tunneling technology , standardized according to RFC4380 . Teredo is part of Windows XP, Vista, and Windows 7; with Miredo , there is also an open-source implementation for Linux and Mac OS X available. Microsoft reports that with Teredo, the chance of a connection between two peers increases from 15% to 84% (PDF link). Our observations are somewhere between 60% and 70%. Teredo is quite sophisticated technology and is a more universal approach. It provides connectivity at the OS level, in contrast to having several applications in use, where each uses its own proprietary traversal technology. Kiril: Could you please elaborate on choosing network technology, having achieved a substantial number of users and collecting usage statistics. Do you know how many active and passive peers you have at any given time? What is the ratio? Wolf: We have solid insight into the state of our P2P network. We know the number of active and passive peers on any given day (using the log from our update server). The active peer ratio is between 60 and 70%. We are also currently working on an improved distributed intraday statistic. (The distributed statistic currently built into the P2P client is not valid anymore for the increased network size. For scalability, every peer has only a limited view of the whole network, which requires more advanced methods for calculating the actual network size.) Kiril: Your search index essentially is a distributed storage system with DHT addressing, right? Wolf: Yes. Kiril: Have you thought about other uses of such technology, beyond search: back-up, private distributed storage, file-sharing, etc. (like Wuala )? In a publication from 2001 (in German), in which I also outlined the idea of a peer-to-peer search engine, this was part of an integrated solution with a P2P Web server, P2P file-sharing and a P2P anonymizer. Due to various legal copyright issues, we are currently not looking into file-sharing. But from a technological standpoint, a distributed storage system is quite universal, from storing a search engine index to attention data, Web pages, instant messages, social network profiles, micro-blogging messages, back-ups and files. Kiril: Could you please share your vision of the mythical “P2P operating system,” now that we already have P2P networking, P2P processing, P2P storage and P2P applications (like search). Wolf: P2P and distributed architectures are a universal principle that the whole Internet is built upon. Unfortunately, distributed technologies like Mail, IRC, Usenet and even independent Web servers are being increasingly replaced by centralized solutions (the cloud, Google Wave, etc.). Despite the obvious short-term convenience, this leads to long-term monopolies and dependencies and makes the Internet infrastructure more vulnerable in terms of reliability and political influence. I believe that a solid, standardized P2P stack integrated in the operating system can fix the broken end-to-end connectivity between users, enabling the use of an endless amount of latent storage, memory, processor cycles and bandwidth. Distributed storage is certainly a core component, as is distributed processing to make more sense of all of the data. On top of this, there should be a distributed programming framework, which enables the development of distributed applications and the distribution and aggregations of tasks in a standardized manner (e.g. the distributed version of MapReduce/Hadoop is part of this). A distributed attention data repository, shared by all applications, but under full user control. There should be resource management that puts the user in full control of the amount of resources she or he would like to dedicate to a particular distributed project—possibly combined with a ratio system and/or virtual currency to maintain a healthy usage to contribution ratio. Distributed identity management and authentication, authorization and access control. This could replace most of the centralized cloud solutions by delivering the same convenience and scalability in a decentralized way. BOINC (the universal distributed processing platform where seti@home runs today) goes partially in that direction. This is partially because the peers contribute, by taking tasks from a centralized server and providing results back to this server. But this system is not fully distributed, nor are the results intended to be used by the peers themselves. Kiril: Do you encounter scalability issues? Do you have any single point of failure resources in your P2P network? How reliable is it—meaning, what percentage of the network could you lose without seriously degrading search quality and performance? Wolf: We have scaled the P2P network in a controlled way. While we have made some scalability-related adjustments to our P2P protocol, the core algorithms proved that there are no inherent scalability limits. Due to our fully distributed architecture, we have no single point of failure. We have twenty-fold redundancy of each item, which replicates automatically if peers leave the network. Only if all 20 copies of the item are removed at the same time would this piece of information be lost. This leads to a mean information lifetime of 120 years under realistic churn (i.e. the peers randomly joining and abandoning the network temporarily or permanently). This is more than sufficient for search, where 50% of the information changes during the year (and is therefore refreshed anyway at a much higher rate). Kiril: Do you think that “mobile P2P” is feasible? What would you say about implementing P2P search (or any other application) on, say, the iPhone? Are mobile terminals ready for P2P? Are cell networks ready? Do they have enough CPU power, etc? Wolf: Today, we distinguish between mobile connectivity and landlines. But I believe this separation will fade away. Device performance, bandwidth and flat-rate pricing structures will become close. While today, processor cycles, memory and bandwidth in mobile phones are too precious for wide use of P2P applications, this will change. Even “walling off” tendencies and restrictive App Store policies will be liberated by regulation or user demand. But much more interesting than bringing file-sharing to the iPhone will be P2P applications that use mobility, possibly combined with GPS, distributed camera/augmented reality and RFID. This will bring P2P technology into completely new application fields. Think of distributed traffic control (peers could be users with iPhones in cars or the cars themselves) or applications to lead crowds of people at large public events or in disaster zones, as well as gaming, distributed weather and earthquake prognosis . Bluetooth could even make this independent of cell networks. Global communication between peers would be asynchronous through moving people . Also, cell network and Bluetooth mashups would be possible. In the near future, we will provide Web access to our P2P Web search for mobile users. They will just be passive users of the resources contributed by active PC users. Kiril: What is your vision of the P2P road map? Apparently, the first “killer P2P application” was file-sharing, Kazaa, then BitTorrent. Given that the next one is search, what would be the next after that? Wolf: As I mentioned, instead of another isolated P2P application, I would like to see P2P built into the OS and Internet stack in a standardized manner. So that an application can benefit from P2P without any specific effort, in the same easy and natural way that applications today use the Internet (HTTP, AJAX and JSON). Then, P2P technology would become ubiquitous and part of almost every application. Every application that uses cloud services today could benefit from such P2P technology. An example would be a distributed platform for micro-blogging services and social networks, heralding the end of walled gardens. But my personal vision is to combine P2P with the next thing after search. Twenty years ago, I wrote a small expert system on my C64 (today, a C64 emulator is on the iPhone!), using Predicate Logic and an Eliza -style natural-language interface. So, you could tell the system, ” All cats have claws. All tigers are cats. ” And then you could ask the system, ” Do tigers have claws? ” And it would answer, ” Yes! ” You could retrieve information and relationships that were not explicitly stored (or that anyone was even aware existed). At that time I had to enter every bit of information manually. Today, almost all information on earth is accessible on the Internet, together with comments and conversation streams. Predicate logic would be supported by fuzzy logic, statistical machine learning and more. Today, known translations are used to translate untranslated text. But this could be much more universal: using known connections in one field to explain unknown correlations in another . Such a system could autonomously formulate queries, combine facts, fill in the missing link in a theory to prove or falsify it. So, I think the next step after search will be reasoning; and in combination with P2P technology and distributed processing, this may bring us a kind of global brain. A brain that not only stores and retrieves information but that is capable of cognition and conclusion at a giant scale. It will discover hidden correlations and unknown facts and will answer questions with answers that cannot be found in any document. You can see, this is much more HAL than the next Google. Kiril: What technical feature of your technology are you most proud of? 100+ patents must something. Wolf: Most crucial has been to ensure both a quick response time and complete results for queries with multiple terms and phrases (only 15% of searches are single keywords) in a completely distribute P2P architecture. For queries with multiple keywords, we eliminated the need for the intersection of huge posting lists across different peers. While we had to invent a lot of things—just because they hadn’t been done before in a way that was required for distributed search—they are not all patented (so, we don’t own 100+ patents). Prior to funding, this would have been impossible financially. Kiril: How is your real-time search related to the P2P search? Does it also run on a distributed network? If so, then how do peers communicate results to the front page of the FAROO website? Wolf: Currently, we use a hybrid architecture. While we are building up our P2P network and use it for general Web search, in parallel we use a central index for the real-time data. The focus on the most recent and popular Web pages keeps the costs moderate. Attention data collected by FAROO peers serves also for real-time discovery and ranking (in addition to analyzing the Twitter stream). But we believe in a holistic approach. Our real-time search will evolve into an integral part of Web search and be fully based on our P2P architecture . There will still be a gateway/proxy server that enables Web access to our P2P network for those users not able or ready to install a P2P client (e.g. for mobile). Discuss

See original here:
Technical Q&A With FAROO Founder

The web isn’t about pages any more. Now it’s about streams, feeds and syndication. As part of our annual Best of Series , below are our picks for the most important RSS and Syndication Technologies of 2009. You can see last year’s list here and most of those remain important services. Only one service makes a repeat appearance this year. It was a very big year for this class of technologies, after a long, sleepy period the Real-Time Web began to cause substantial disruptions over the last 12 months. Check out our list below and let us know if we’ve missed anything important or who your picks might be for next year. Sponsor This is the fifth in our series on top products of 2009: Top 10 Mobile Web Products of 2009 Top 10 Consumer Web Apps of 2009 Top 10 Semantic Web Products of 2009 Top 10 International Web Products of 2009 Facebook has 350 million users today. Just 12 months ago there were a mere 140 million Facebook users . A syndicated stream is the default view in Facebook, meaning that 210 million more people have been introduced to this paradigm by Facebook in 2009. That’s a powerful cultural change. Twitter may not be anywhere near the size of Facebook, nor growing as fast, but for tens of millions of people, 2009 was a year they got comfortable with streams, lists (just like cute little OPML files!) and soon geolocation data – thanks to Twitter. Echo, from JS-Kit is a reverse syndication service for distributed social media conversations. It brings back tweets and other mentions to the page they refer to. The service is growing fast and becoming more sophisticated every week. New features come so fast and furious that it’s overwhelming but the end result is an experience that brings the dispersed social web back together again. Fever is a gorgeous new RSS reader that costs $30 and lives on your own server. It’s got a very interesting system for ranking hot stories by your own criteria – we just wish we could change the timeframe so that ranking was for every 2 or 3 hours, not per day. Fever looks great and works wonderfully on the iPhone. If people ask you what good web-based alternatives there are to Google Reader, Fever is a good place to start looking. PubSubHubbub (and RSSCloud ) are two feed formats for the real-time web. PubSubHubbub is method for pushing real-time updates from a publisher, to a hub and then to all subscribed parties – immediately. RSSCloud is a similar technology that originated years ago as a part of the RSS spec. These are the protocols that a whole new era of user and developer experience on the web will be built on. Superfeedr is a new service powering millions of real-time feeds. It’s a transformer, from lots of different formats into real-time feeds in PubSubHubbub or XMPP. It’s like FeedBurner for the real-time web. Tweetdeck (and Seesmic ) are the market’s leading stream readers. They are tools for reading and writing to Twitter, Facebook, MySpace, LinkedIn and someday other social network streams. There are lots of innovative stream readers on the market, from the beautiful Skimmer to the Inspector-Gadget style Favit , but Tweetdeck is the clear market leader. It’s in a perpetual back-and-forth battle of the sweet features with Seesmic. Both are dramatically changing the way users experience the flowing social web. Honorable Mentions: Feedly Twingly Twitterfeed Public Radio Tuner Regator Collected.info Postrank just keeps getting smarter. This social media analytics service tracks distributed conversation regarding blogs and feeds and scores items based on the relative engagement of those conversations. The usefulness of this service just doesn’t stop and the company’s movement into large-publisher analytics and APIs this year should bode well for customers, developers and consumers. Postrank is the only service on this list that was also on 2008’s list. ActivityStreams is a proposed standard way to markup user activity data in social networks. If everyone adopted the standard, then streams of data would be interoperable, we could see what friends on other networks were doing and we wouldn’t be locked-in to the big networks because little innovators could provide tools for conversation. So far Facebook, MySpace, Netflix, Sun Microsystems and more are working hard at making this a reality. 2009 was a big year for ActivityStreams, right down to last week’s announcement that a feed normalization API was released by startup Cliqset . The Breaking News Online iPhone App is the best remnant of a fabulous story that’s changed dramatically in recent weeks. BNO is a news organization that’s so fast in breaking news from around the world that the Red Cross watches them for disaster news and MSNBC syndicates their stories. Unfortunately, the company owned by now 19 year old Michael van Poppel sold control over its wildly popular Twitter account to MSNBC this Winter, but the iPhone app remains a very valuable resource. BNO’s research and original reporting is definitely one of the biggest stories in syndication of 2009 and its iPhone app is a must-have. Discuss

See original here:
Top 10 RSS & Syndication Technologies of 2009

Two years and a month after announcing that it would launch a more professional-looking developer platform than the wildly successful one at Facebook, LinkedIn today finally opened up a series of application programming interfaces for other companies to build on top of. Make no mistake about it, though – there’s some good news and there’s some bad news. LinkedIn holds an incredibly useful body of data about its users – not just because of the relatively high net worth it brags about its users having but because employment information is a very useful way to put a person in context on the web. That data is now available for an ecosystem of other developers to incorporate; TweetDeck, Posterous, Ribbit and several other applications already have. Sponsor The Good News It’s easy to get started. After two years of waiting, unreplied emails and heartbreak – developers should now be able to get an API key within minutes and start building on the LinkedIn platform. That’s great news and not something that could have been taken for granted. The API allows search. That’s great because with a little disambiguation done on the client side you can find the LinkedIn accounts of people you’re connected to on other networks. Unfortunately, no one is doing exactly that yet – but isn’t that the biggest value proposition here? I see a person on Twitter, on Facebook, on some other social network and I want to see what they do for a living. Let the app collect and expose that data from LinkedIn! Disambiguation of people with the same name and privacy limitations regarding who gets to see who’s information are both complicating factors. The coolest use of the search API we’ve seen so far is Salim Ishmael’s Knx.to . That service is limited to your own connections so far, but it’s definitely a keeper. The API uses OAuth . That means that 3rd parties can offer fast, secure, standardized authentication into your LinkedIn user account. That’s great. Activity updates are now parsable by type. The API allows developers to pull in just one type of the many updates a person gets on LinkedIn. Will someone please build an app that just shows me when my contacts change jobs and leaves out all the status messages, friend connections and other cruft? That kind of granular control has a lot of potential and is reminiscent of the vision behind the proposed user activity data protocol Activity Streams . And Now For the Bad News… The first use-cases make it look like LinkedIn is trying to be Twitter. Tweetdeck and Posterous are the most high-profile early adopters of the new API; Tweetdeck will give you a LinkedIn column (too bad LinkedIn contacts can’t be integrated into other columns) and Posterous will let you publish links to updates on that platform over to your LinkedIn contacts’ streams. Jobdash looks like Tweetdeck just for LinkedIn and job-hunting, but it doesn’t yet offer features like limited display of notifications by type – it’s just a big stream of updates. LinkedIn is not Twitter! LinkedIn’s Adam Nash told us this morning that he loves the Twitter and Twitter-like integrations but “integrating messaging isn’t the goal, there’s a wide range of business applications that will benefit from it. Twitter is hot so people are jumping to that but there are far more compelling business cases.” Two years after the business-oriented platform was announced tiny Tweetdeck was just so hot it out-maneuvered all the business applications that could have been built to showcase? I don’t buy it. Just like the formal partnership between Twitter and LinkedIn earlier this month , I worry that this API is built with marketing, promotion and broadcast functions best served. Terms and Conditions are unclear, restrictive and changing. The API terms say that you can’t build applications that compete with LinkedIn. API management service Mashery CEO Oren Michels (disclosure: RWW sponsor) had this in response to say: “It appears that you can’t create a new experience around LinkedIn, an iPhone app for example. You might create some interesting bolt-ons to other services that might drive users to linkedin.com – but that’s a very 5 years-ago approach to an API.” “The signal from this is that they aren’t encouraging developers to take the social graph and deep knowledge of peoples’ professional lives and create new UIs for interacting with LinkedIn because they are explicitly concerned about competition,” Michels said. “LinkedIn has amazing assets and a great business model – get out of the UI business!” Likewise several developers have expressed concern around the commercial limitations on the API. LinkedIn’s Nash clarified with us that those terms simply prohibit charing people extra money for access to the free LinkedIn service and building an advertising network on top of LinkedIn profile data because of privacy concerns. Finally, the terms of the API aren’t very clear. Nash said they were a work in progress and likely to be changed. Some of those changes appear to have been made in the 3 hours since the API launched . Likewise, Michels points out that rate limits on accessing the API aren’t made explicit – only that there will be rate limits and that a developer can email LinkedIn to request a personal expansion of their limit. Not playing nice with others: LinkedIn is exposing what it calls an Activity Stream, but it’s not at all related to the standardized format that Facebook, MySpace, Netflix and others are now publishing. LinkedIn publishes some Microformats but has been entirely absent from the wide-ranging community discussion of Activity Streams formats, we’re told. Michaels may have said it best: “There are some really smart people over there at LinkedIn. If this is what we waited 2 and a half years for, it’s a bit disappointing.” It is a bit, but not entirely disappointing. We look forward to seeing how the platform evolves and what kinds of applications are built on top of it. The web has been waiting a long time for a LinkedIn platform – now let’s see what happens. Discuss

Read more:
LinkedIn Finally Opens Platform: The Good & Bad News