The majority of motorists (66%) would be uncomfortable travelling in a driverless car at 70mph, according the Institution of Mechanical Engineers.
Of the 2,053 asked, younger people tend to be more accepting of the technology - with 45% of 25-36 year olds saying they would be comfortable in a 70mph driverless car.
With only 13% for 65-74 year olds and 8% for the over 75s.
Women tended to be more cautious about the technology, with 72% saying they would be uncomfortable compared to 60% for men.
The survey found that 50% of the public think that humans are better drivers than computers/cars, despite the fact that 90% of UK road accidents are the result of driver error.
While the survey also showed that there is a reluctance from the public to allow people who are sight-impaired to be the sole occupant of a driverless car - with just 23% saying this should be allowed.
There was also very little acceptance for people who are intoxicated being responsible for a driverless car, with just 12% saying this would be acceptable.
Institution of Mechanical Engineers’ head of transport Philippa Oldham said: “The benefits of driverless technology are huge.
"Not only could the technology help save hundreds of lives, but there are estimates that the overall UK economic benefit could be as much as £51 billion a year due to fewer accidents, improved productivity and increased trade.
“The Government and manufacturers have big ambitions for the future of driverless cars, but there is still a long way to go in terms of public approval.
“This study shows that the majority of people would be uncomfortable being a passenger in a driverless car travelling at speed and that older people and women are particularly wary.
“Driverless technology has been touted as improving accessibility for all, but the survey showed strong reluctance for a sight-impaired person to be the sole occupant of a driverless car.
“Given the huge benefits to this technology it is vital for Government and manufacturers develop a public campaign with more demonstrations and user trials to build awareness and trust in this technology.
"In addition to the driverless lorries trials, another idea could be for driverless technology demonstrators to be rolled out on city roads, allowing people the see and get first-hand experience of these vehicles in action.
"These cars could be a particularly eye-catching colour, and their presence on busy city roads could help make people more aware of, accustomed to and accepting of the technology.”