Alan Manning, who chairs the Migration Advisory Committee, said fruit and vegetable growers would probably “go backwards”.
But it would not be the “end of the earth” for the UK economy as a whole.
The National Farmers Union has warned that without seasonal workers crops like strawberries will go unpicked.
The government has said high-skilled workers would be prioritised after Brexit, with no preferential treatment for people from the EU compared with those from the rest of the world.
The policy is based on recommendations drawn up by Mr Manning’s committee, which have also been backed by the Labour Party.
Mr Manning said the rules on skilled workers – which say they must earn more than £30,000 a year to get a visa – might be relaxed for a “tiny” number of highly-skilled, low-paid individuals, such as musicians or dancers.
But he said many firms who had benefited from the influx of low-skilled East and Central European workers over the last 14 years would “find life a bit harder” under the proposed new regime.
“Most of them are not musicians, these are people working in warehouses and food manufacturing, in hospitality and so on,” he told the Lords EU home affairs sub-committee.
“Our view would be they have had a tailwind since 2004, which those sectors would understandably want to continue, but it’s not necessarily clear that that is in the interests of the wider economy and society.”
He said seasonal agricultural workers were a special case, because they all came from EU countries (99% according to the Office for National Statistics).
Mr Manning told the Lords committee there was “no realistic prospect” of that work being done by British people.
The government has launched a pilot scheme to allow farmers to recruit 2,500 non-EU migrants to help the industry adjust – a move welcomed by the National Farmers Union, which has been warning of crops being left to rot in the ground.
Mr Manning said the fruit and veg growing sector had “expanded a lot” since the so-called A8 countries, including Poland, joined the EU, with an “extraordinary” increase in the amount of land given over to crops like asparagus, strawberries and other seasonal produce.
“If you cut off the access to that labour those sectors would find it much harder,” he said.
“To some extent, it’s quite likely that they would contract.”
He said agriculture as a whole was a low productivity sector, about 40% of the national average.
“So if you are wanting to make the UK a high wage, high productivity economy, which we generally are, it’s not clear that making life very easy for agriculture – giving them privileged access to labour – is a way to achieve that.
“That really wouldn’t be the end of the earth for the country as a whole. Obviously the NFU are not going to be very enthusiastic about it.”
The Migration Advisory Committee has suggested forcing farmers to pay a higher minimum wage in order to encourage increases in productivity – or charging them for every additional foreign worker they employ.