At the top of a VideoSurf results page for ‘Mad Men,’ users can search for clips featuring specific characters.

This week I tested VideoSurf.com, a site that claims to be the first to search videos by “seeing” images that appear in these videos. The company says its technology can analyze a clip’s visual content, as well as its metadata — especially when searching for people. VideoSurf has analyzed and categorized more than 12 billion visual moments on the Web to understand who the most important characters and scenes are in a video, and it uses this knowledge to sort clips according to relevancy.

Search results on VideoSurf spread out videos in a filmstrip-like format, distinguishing one scene from the next. Users can choose an option to show only faces, which helps if you’re looking for a specific person in a long video or movie. And when looking at videos from certain sources, you can select a scene from the filmstrip and jump ahead to that scene rather than sit through the entire clip.

When it works, VideoSurf is one of those technologies that make you wonder why someone didn’t think of it sooner. The site aggregates content from about 60 sources, including YouTube, CNN Video, Hulu, ESPN and Comedy Central, and a sorting tool weeds out unwanted results like the irksome slideshows that are labeled as videos. VideoSurf can find videos on all kinds of subjects, but it really shines when it finds well-known people.

But VideoSurf has some rough edges and doesn’t always work as it should. In its defense, the site is still in its public beta, or trial, stage, and plans to be full-blown by early next year. Right now, one of its best features, the ability to jump ahead to specific scenes, works with video from only a handful of sources including YouTube, MetaCafe, DailyMotion and Google (GOOG) Video. Videos from Hulu.com confusingly allow jumping ahead only from certain screens.

Additionally, I came across a couple of videos that were no longer available, though they were listed in search results. And a customizable VideoSurf home page for users with accounts on the site saves searches but not specific clips; VideoSurf plans to fix this next week by adding a favorites page where users can store and share favorite videos with others.

Still, I really grew to like VideoSurf’s clear way of displaying content that would be otherwise buried within videos. Rather than trying to guess a video’s contents by looking at a single representative image, VideoSurf’s filmstrip views showed me exactly what I’d be watching. In many cases, I viewed a video I might not have otherwise watched because its filmstrip showed shots of scenes that looked interesting.

On the left-hand side of the search-results page, VideoSurf users can narrow results according to Content Type, Categories and Video Sources to see just what they’re looking for — or, often more important, what they’re not looking for. Content Type, for example, includes slideshows, Web series, full television episodes and full movies; a search can include only videos in a particular category (say, slideshows) or exclude that category altogether by unmarking the box beside it.

Most search-results pages include tiled still images at the top representing the characters in the videos. By selecting one of these characters, users can refine search results to show only videos with that character. For example, I typed the title of a favorite television show, “Brothers and Sisters,” into the search box and saw the names and images of seven actors on the show at the top of the screen. I selected Sally Field and was redirected to results of videos featuring only the mother she plays on the show.

I used VideoSurf to search for Beyonce’s “Single Ladies” music video, and then changed the date parameters to find only videos posted this week. This retrieved a Saturday Night Live skit in which the pop singer spoofs her own video with help from three men in tights — including Justin Timberlake. While the SNL skit ran, a list of related videos appeared in a column on the right, including clips of J.T.’s past SNL skits.

Occasionally, annotations appear on videos, but these come from the source — not VideoSurf. If overlaid text appears on YouTube videos, it can be turned off using an icon in the bottom right of the YouTube screen. Video-sharing sites that use introductory pages such as pre-rolls before each video will still show those pages.

VideoSurf makes it easy to send specific clips of videos to friends. I did so by selecting a Share option and adjusting slide bars to trim the clip to start and end at scenes I preferred. Clips shared with friends via email are sent with the VideoSurf filmstrip, giving others the ability to also know what the video will include so that they, too, can discern whether or not they want to watch it.

Clips can be shared on social-networking sites like del.icio.us, MySpace and Facebook, though VideoSurf’s helpful filmstrip didn’t show up on these sites like it did in emails.

I also tested an add-on for the Mozilla Firefox browser called Greasemonkey that works with VideoSurf. When installed, this displays VideoSurf’s helpful filmstrip beneath search results from Google Video, YouTube, Yahoo (YHOO) or CBS.com (CBS). Once installed, filmstrips illustrating important scenes appear along with the normal text results for videos, and some of the filmstrips enable jumping ahead to specific scenes. This somewhat techie Greasemonkey extension can save people the extra step of making a separate visit to VideoSurf.com to watch a specific clip.

VideoSurf uses smart technology that can save people the aggravation of watching videos that aren’t what they appear to be. Since so much Web content now includes videos, a visual search tool that can better assess videos like VideoSurf is a good idea. When this site improves its now-flaky ability to jump ahead to specific scenes in videos, it will be even more valuable.

Edited by Walter S. Mossberg

• Email us at mossbergsolution@wsj.com. Find this and other columns and videos online free at the All Things Digital Web site: http://walt.allthingsd.com